[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:22:51 dist-test-slave-q03f krb5kdc[32608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:22:51 dist-test-slave-q03f krb5kdc[32608](info): set up 2 sockets
Sep 28 06:22:51 dist-test-slave-q03f krb5kdc[32608](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 06:22:52.698510 32601 mini_kdc.cc:121] Time spent starting KDC: real 2.536s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:22:52 dist-test-slave-q03f krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:22:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:22:52.834348 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:42261
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:42261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:52.924597 32621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:52.924839 32621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:52.924863 32621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:52.926034 32621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:22:52.926071 32621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:52.926086 32621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:52.926100 32621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:22:52.926115 32621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:22:52.927563 32621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:42261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:42261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:52.927811 32621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:52.928025 32621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:22:52.940995 32621 server_base.cc:1034] running on GCE node
W20260928 06:22:52.941013 32627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:52.940968 32629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:52.940958 32626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:52.941532 32621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:52.941903 32621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:52.943137 32621 hybrid_clock.cc:648] HybridClock initialized: now 1790576572943103 us; error 57 us; skew 500 ppm
Sep 28 06:22:52 dist-test-slave-q03f krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576572, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:52.946112 32621 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:22:52.946651 32621 webserver.cc:460] Webserver started at http://127.31.214.126:37819/ using document root <none> and password file <none>
I20260928 06:22:52.946918 32621 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:52.946995 32621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:52.947167 32621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:52.948450 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "e67ef77201dd44398f8007c46f965236"
format_stamp: "Formatted at 2026-09-28 06:22:52 on dist-test-slave-q03f"
server_key: "3af1fd2940f147f2b84409811dc29a72"
server_key_iv: "71099561b43a4959b599f9d26d57d4e5"
server_key_version: "encryptionkey@0"
I20260928 06:22:52.949064 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "e67ef77201dd44398f8007c46f965236"
format_stamp: "Formatted at 2026-09-28 06:22:52 on dist-test-slave-q03f"
server_key: "3af1fd2940f147f2b84409811dc29a72"
server_key_iv: "71099561b43a4959b599f9d26d57d4e5"
server_key_version: "encryptionkey@0"
I20260928 06:22:52.950744 32621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 06:22:52.951712 32636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:52.952101 32621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:52.952251 32621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "e67ef77201dd44398f8007c46f965236"
format_stamp: "Formatted at 2026-09-28 06:22:52 on dist-test-slave-q03f"
server_key: "3af1fd2940f147f2b84409811dc29a72"
server_key_iv: "71099561b43a4959b599f9d26d57d4e5"
server_key_version: "encryptionkey@0"
I20260928 06:22:52.952349 32621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:52.965341 32621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:52.968196 32621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:52.968345 32621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:52.972689 32621 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:42261
I20260928 06:22:52.972725 32688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:42261 every 8 connection(s)
I20260928 06:22:52.973112 32621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:22:52.974179 32689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:52.977607 32689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236: Bootstrap starting.
I20260928 06:22:52.978351 32689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:52.978659 32689 log.cc:824] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:52.979473 32689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236: No bootstrap required, opened a new log
I20260928 06:22:52.980465 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 32621
I20260928 06:22:52.980614 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:22:52.981419 32601 external_mini_cluster.cc:1428] Setting key 10dbd7036adb6dd8926e23ab37e8b058
I20260928 06:22:52.981825 32689 raft_consensus.cc:348] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67ef77201dd44398f8007c46f965236" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42261 } }
I20260928 06:22:52.982101 32689 raft_consensus.cc:374] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:52.982185 32689 raft_consensus.cc:729] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67ef77201dd44398f8007c46f965236, State: Initialized, Role: FOLLOWER
I20260928 06:22:52.982347 32689 consensus_queue.cc:260] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67ef77201dd44398f8007c46f965236" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42261 } }
I20260928 06:22:52.982465 32689 raft_consensus.cc:388] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:22:52.982508 32689 raft_consensus.cc:482] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:22:52.982569 32689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:52.984303 32689 raft_consensus.cc:504] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67ef77201dd44398f8007c46f965236" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42261 } }
I20260928 06:22:52.984459 32689 leader_election.cc:302] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e67ef77201dd44398f8007c46f965236; no voters:
I20260928 06:22:52.984683 32689 leader_election.cc:288] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:22:52.984732 32694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:22:52.984939 32694 raft_consensus.cc:686] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [term 1 LEADER]: Becoming Leader. State: Replica: e67ef77201dd44398f8007c46f965236, State: Running, Role: LEADER
I20260928 06:22:52.985056 32689 sys_catalog.cc:564] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:22:52.985095 32694 consensus_queue.cc:237] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67ef77201dd44398f8007c46f965236" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42261 } }
I20260928 06:22:52.985441 32694 sys_catalog.cc:455] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e67ef77201dd44398f8007c46f965236. Latest consensus state: current_term: 1 leader_uuid: "e67ef77201dd44398f8007c46f965236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67ef77201dd44398f8007c46f965236" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42261 } } }
I20260928 06:22:52.985520 32694 sys_catalog.cc:458] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [sys.catalog]: This master's current role is: LEADER
I20260928 06:22:52.985625 32696 sys_catalog.cc:455] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e67ef77201dd44398f8007c46f965236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67ef77201dd44398f8007c46f965236" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42261 } } }
I20260928 06:22:52.985738 32696 sys_catalog.cc:458] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236 [sys.catalog]: This master's current role is: LEADER
I20260928 06:22:52.985884 32700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:22:52.986524 32700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:22:52.988444 32700 catalog_manager.cc:1347] Generated new cluster ID: b7b8ff3426654dbeae497038f7813dab
I20260928 06:22:52.988502 32700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:22:52 dist-test-slave-q03f krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:53.000367 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:52.982636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.31.214.126:42261)
0928 06:22:52.983541 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:52.983549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:52.983578 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:22:52.984964 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:52.984971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:52.985031 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:52.985171 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:52.987593 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:52.988884 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:52.990375 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:52.990956 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:52.991077 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:52.995369 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:52.995374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:52.995375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:52.995395 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:52.998095 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:52.998683 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:52.998686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:52.998687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:52.998827 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:52.999144 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:52.999145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:52.999145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:52.999479 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:52.999638 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:52.999946 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 06:22:53.000090 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":96,"threads_started":1}
I20260928 06:22:53.001636 32700 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:22:53.002178 32700 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:22:53.016484 32700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e67ef77201dd44398f8007c46f965236: Generated new TSK 0
I20260928 06:22:53.016799 32700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:22:53.050252 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:42261
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:53.135731 32715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:53.135953 32715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:53.135978 32715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:53.137171 32715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:53.137216 32715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:53.137244 32715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:22:53.138850 32715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:42261
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:53.139115 32715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:53.139394 32715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:53.144729 32720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:53.144798 32721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:53.144729 32723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:53.145011 32715 server_base.cc:1034] running on GCE node
I20260928 06:22:53.145213 32715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:53.145421 32715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:53.146586 32715 hybrid_clock.cc:648] HybridClock initialized: now 1790576573146596 us; error 59 us; skew 500 ppm
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:53.149556 32715 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:22:53.150105 32715 webserver.cc:460] Webserver started at http://127.31.214.65:37489/ using document root <none> and password file <none>
I20260928 06:22:53.150333 32715 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:53.150380 32715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:53.150518 32715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:53.151589 32715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "fad5b410f113477694c5ff592b853d1d"
format_stamp: "Formatted at 2026-09-28 06:22:53 on dist-test-slave-q03f"
server_key: "1e96b6226399b721cbee0391c1d06334"
server_key_iv: "410cbdeb1ac0b7d8f14364b7a53f398f"
server_key_version: "encryptionkey@0"
I20260928 06:22:53.151959 32715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "fad5b410f113477694c5ff592b853d1d"
format_stamp: "Formatted at 2026-09-28 06:22:53 on dist-test-slave-q03f"
server_key: "1e96b6226399b721cbee0391c1d06334"
server_key_iv: "410cbdeb1ac0b7d8f14364b7a53f398f"
server_key_version: "encryptionkey@0"
I20260928 06:22:53.153354 32715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:22:53.154296 32730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:53.154584 32715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:53.154702 32715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "fad5b410f113477694c5ff592b853d1d"
format_stamp: "Formatted at 2026-09-28 06:22:53 on dist-test-slave-q03f"
server_key: "1e96b6226399b721cbee0391c1d06334"
server_key_iv: "410cbdeb1ac0b7d8f14364b7a53f398f"
server_key_version: "encryptionkey@0"
I20260928 06:22:53.154805 32715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:53.176954 32715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:53.179263 32715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:53.179422 32715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:53.179700 32715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:22:53.180058 32715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:53.180095 32715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:53.180122 32715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:53.180169 32715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:53.186815 32715 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:42183
I20260928 06:22:53.186939 375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:42183 every 8 connection(s)
I20260928 06:22:53.187261 32715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:53.195986 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 32715
I20260928 06:22:53.196128 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:22:53.196312 32601 external_mini_cluster.cc:1428] Setting key 34bc9c0849b39d0be1c429bbebfa491e
I20260928 06:22:53.197551 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:53.188091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:41199 (local address 127.31.214.126:42261)
0928 06:22:53.188160 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:53.188162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:53.188388 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:22:53.188943 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:53.188944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:53.188954 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:53.189018 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:53.190048 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.190610 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.191682 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.191885 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.191946 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:53.194331 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:53.194332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:53.194332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:53.194339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:53.195842 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.196395 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.196396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.196396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.196465 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.196824 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.196825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.196826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.196959 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:53.197033 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:53.197353 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 06:22:53.197457 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:22:53.197767 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:53.188055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:42261 (local address 127.31.214.65:41199)
0928 06:22:53.188254 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:53.188297 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:53.188855 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:53.189113 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:53.189117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:53.189706 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:53.189921 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:53.189926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.190725 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.190726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:53.191586 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:53.191588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.191976 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.191977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:53.192088 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:53.192618 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:53.192626 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:53.194233 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:53.195948 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:53.195957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:53.195960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:53.196298 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:53.196549 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:53.196550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:53.196551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:53.196743 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:53.197037 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:53.197044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:53.197208 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 06:22:53.197330 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 06:22:53.197505 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260928 06:22:53.198349 376 heartbeater.cc:347] Connected to a master server at 127.31.214.126:42261
I20260928 06:22:53.198451 376 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:53.198686 376 heartbeater.cc:511] Master 127.31.214.126:42261 requested a full tablet report, sending...
I20260928 06:22:53.199477 32653 ts_manager.cc:194] Registered new tserver with Master: fad5b410f113477694c5ff592b853d1d (127.31.214.65:42183)
I20260928 06:22:53.200528 32653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:41199
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:22:53.226068 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:42261
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:53.318493 381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:53.318708 381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:53.318732 381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:53.319846 381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:53.319880 381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:53.319907 381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:22:53.321410 381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:42261
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:53.321604 381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:53.321784 381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:53.326933 386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:53.326905 389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:53.326957 387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:53.327226 381 server_base.cc:1034] running on GCE node
I20260928 06:22:53.327432 381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:53.327677 381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:53.328862 381 hybrid_clock.cc:648] HybridClock initialized: now 1790576573328843 us; error 28 us; skew 500 ppm
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:53.331751 381 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:22:53.332273 381 webserver.cc:460] Webserver started at http://127.31.214.66:34041/ using document root <none> and password file <none>
I20260928 06:22:53.332505 381 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:53.332584 381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:53.332736 381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:53.333854 381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "9f91a039912a4832a1917e13d63047b9"
format_stamp: "Formatted at 2026-09-28 06:22:53 on dist-test-slave-q03f"
server_key: "ec6a9d81fd1f022bc6ec95dfcea484fd"
server_key_iv: "4b0d626cfc36ff6c581be5a961f8922a"
server_key_version: "encryptionkey@0"
I20260928 06:22:53.334307 381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "9f91a039912a4832a1917e13d63047b9"
format_stamp: "Formatted at 2026-09-28 06:22:53 on dist-test-slave-q03f"
server_key: "ec6a9d81fd1f022bc6ec95dfcea484fd"
server_key_iv: "4b0d626cfc36ff6c581be5a961f8922a"
server_key_version: "encryptionkey@0"
I20260928 06:22:53.335733 381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:53.336555 396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:53.336750 381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:53.336866 381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "9f91a039912a4832a1917e13d63047b9"
format_stamp: "Formatted at 2026-09-28 06:22:53 on dist-test-slave-q03f"
server_key: "ec6a9d81fd1f022bc6ec95dfcea484fd"
server_key_iv: "4b0d626cfc36ff6c581be5a961f8922a"
server_key_version: "encryptionkey@0"
I20260928 06:22:53.336944 381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:53.351526 381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:53.354229 381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:53.354408 381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:53.354699 381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:22:53.355046 381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:53.355103 381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:53.355149 381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:53.355175 381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:53.362365 381 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:43065
I20260928 06:22:53.362414 509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:43065 every 8 connection(s)
I20260928 06:22:53.362774 381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:53.371426 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 381
I20260928 06:22:53.371546 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:22:53.371716 32601 external_mini_cluster.cc:1428] Setting key c640b7abd7352801ecc6bff5e48eaed7
I20260928 06:22:53.373488 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:53.363573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:56521 (local address 127.31.214.126:42261)
0928 06:22:53.363647 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:53.363649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:53.363885 (+ 236us) server_negotiation.cc:407] Connection header received
0928 06:22:53.364467 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:53.364468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:53.364481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:53.364529 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:53.365709 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.366341 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.367433 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.367665 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.367751 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:53.370191 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:53.370193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:53.370193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:53.370201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:53.371827 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.372373 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.372374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.372374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.372448 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.372771 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.372772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.372772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.372891 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:53.372947 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:53.373336 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 06:22:53.373420 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:22:53.373826 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:53.363571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:42261 (local address 127.31.214.66:56521)
0928 06:22:53.363744 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:53.363782 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:53.364353 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:53.364703 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:53.364706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:53.365286 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:53.365595 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:53.365600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.366458 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.366459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:53.367290 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:53.367291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.367758 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.367760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:53.367895 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:53.368472 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:53.368481 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:53.370063 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:53.371948 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:53.371955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:53.371957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:53.372257 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:53.372527 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:53.372528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:53.372528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:53.372693 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:53.372987 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:53.372994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:53.373188 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 06:22:53.373310 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 06:22:53.373502 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260928 06:22:53.374399 510 heartbeater.cc:347] Connected to a master server at 127.31.214.126:42261
I20260928 06:22:53.374518 510 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:53.374717 510 heartbeater.cc:511] Master 127.31.214.126:42261 requested a full tablet report, sending...
I20260928 06:22:53.375172 32653 ts_manager.cc:194] Registered new tserver with Master: 9f91a039912a4832a1917e13d63047b9 (127.31.214.66:43065)
I20260928 06:22:53.375598 32653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:56521
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:22:53.402267 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:42261
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:53.489476 515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:53.489702 515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:53.489729 515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:53.490986 515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:53.491035 515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:53.491066 515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:22:53.492511 515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:42261
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:53.492722 515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:53.492923 515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:53.498234 521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:53.498239 520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:53.498266 523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:53.498773 515 server_base.cc:1034] running on GCE node
I20260928 06:22:53.498971 515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:53.499315 515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:53.500604 515 hybrid_clock.cc:648] HybridClock initialized: now 1790576573500591 us; error 33 us; skew 500 ppm
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:53.503934 515 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:22:53.504371 515 webserver.cc:460] Webserver started at http://127.31.214.67:32931/ using document root <none> and password file <none>
I20260928 06:22:53.504714 515 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:53.504796 515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:53.504937 515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:53.506114 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "fa55b6f6b84c4e8789b983068d3d7301"
format_stamp: "Formatted at 2026-09-28 06:22:53 on dist-test-slave-q03f"
server_key: "9fa6bd12ea9e32f6f89eeb8950e1f559"
server_key_iv: "20b2a45d7019facfcd11a87db152892c"
server_key_version: "encryptionkey@0"
I20260928 06:22:53.506496 515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "fa55b6f6b84c4e8789b983068d3d7301"
format_stamp: "Formatted at 2026-09-28 06:22:53 on dist-test-slave-q03f"
server_key: "9fa6bd12ea9e32f6f89eeb8950e1f559"
server_key_iv: "20b2a45d7019facfcd11a87db152892c"
server_key_version: "encryptionkey@0"
I20260928 06:22:53.507933 515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:22:53.508800 530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:53.508994 515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:53.509101 515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "fa55b6f6b84c4e8789b983068d3d7301"
format_stamp: "Formatted at 2026-09-28 06:22:53 on dist-test-slave-q03f"
server_key: "9fa6bd12ea9e32f6f89eeb8950e1f559"
server_key_iv: "20b2a45d7019facfcd11a87db152892c"
server_key_version: "encryptionkey@0"
I20260928 06:22:53.509191 515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:53.520713 515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:53.523062 515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:53.523289 515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:53.523552 515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:22:53.523892 515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:53.523948 515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:53.523994 515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:53.524019 515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:53.531814 515 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:36619
I20260928 06:22:53.531898 643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:36619 every 8 connection(s)
I20260928 06:22:53.532246 515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:22:53.537832 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 515
I20260928 06:22:53.537933 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:22:53.538137 32601 external_mini_cluster.cc:1428] Setting key b58c9738c0b418dcd2b4c1a37acbdf73
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:53.543027 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:53.533097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:38057 (local address 127.31.214.126:42261)
0928 06:22:53.533196 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:53.533198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:53.533395 (+ 197us) server_negotiation.cc:407] Connection header received
0928 06:22:53.533876 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:53.533877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:53.533888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:53.534033 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:53.535053 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.535658 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.536738 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.536931 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.536992 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:53.539702 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:53.539704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:53.539704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:53.539712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:53.541375 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.541852 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.541853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.541853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.541919 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.542338 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.542339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.542340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.542467 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:53.542529 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:53.542853 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 06:22:53.542932 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:22:53.543148 646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:53.533039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:42261 (local address 127.31.214.67:38057)
0928 06:22:53.533237 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:53.533282 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:53.533779 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:53.534134 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:53.534138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:53.534728 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:53.534938 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:53.534943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.535770 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.535771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:53.536633 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:53.536635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.537021 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.537022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:53.537174 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:53.537814 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:53.537826 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:53.539551 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:53.541474 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:53.541481 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:53.541483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:53.541753 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:53.542057 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:53.542057 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:53.542058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:53.542247 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:53.542539 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:53.542544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:53.542696 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 06:22:53.542793 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:22:53.542930 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":123,"threads_started":1}
I20260928 06:22:53.543757 644 heartbeater.cc:347] Connected to a master server at 127.31.214.126:42261
I20260928 06:22:53.543851 644 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:53.544023 644 heartbeater.cc:511] Master 127.31.214.126:42261 requested a full tablet report, sending...
I20260928 06:22:53.544481 32653 ts_manager.cc:194] Registered new tserver with Master: fa55b6f6b84c4e8789b983068d3d7301 (127.31.214.67:36619)
I20260928 06:22:53.544898 32653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:38057
I20260928 06:22:53.550477 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:22:53.574460 651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:53.566478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.31.214.65:42183)
0928 06:22:53.566693 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:53.566701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:53.566715 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:22:53.566748 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:53.566750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:53.566769 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:53.566865 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:53.567470 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.568216 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.568938 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.569316 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.569376 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:53.571670 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:53.571679 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:53.571679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:53.571686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:53.573215 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.573554 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.573557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.573557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.573616 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.573820 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.573820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.573821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.574022 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:53.574139 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:53.574269 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:22:53.574337 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
W20260928 06:22:53.574959 32758 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42146
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:22:53.599948 651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:53.591944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.31.214.65:42183)
0928 06:22:53.592086 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:53.592088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:53.592103 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:22:53.592132 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:53.592133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:53.592147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:53.592208 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:53.592934 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.593501 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.594423 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:53.594678 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:53.594744 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:53.597135 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:53.597136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:53.597137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:53.597145 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:53.598659 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.599113 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.599115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.599115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.599179 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:53.599425 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:53.599426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:53.599426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:53.599547 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:53.599672 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:53.599798 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:22:53.599878 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 06:22:53.600751 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 32715
I20260928 06:22:53.606303 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 381
I20260928 06:22:53.612495 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 515
I20260928 06:22:53.617225 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 32621
2026-09-28T06:22:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3498 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[658](info): set up 2 sockets
Sep 28 06:22:53 dist-test-slave-q03f krb5kdc[658](info): commencing operation
krb5kdc: starting...
W20260928 06:22:54.721585 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:22:54 dist-test-slave-q03f krb5kdc[658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:22:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:22:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:22:54.850589 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:36633
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:54.933079 671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:54.933346 671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:54.933382 671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:54.934587 671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:22:54.934661 671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:54.934685 671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:54.934706 671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:22:54.934736 671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:22:54.936261 671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:36633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:54.936549 671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:54.936811 671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:54.941929 679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:54.941989 671 server_base.cc:1034] running on GCE node
W20260928 06:22:54.941937 677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:54.941937 676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:54.942397 671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:54.942719 671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:54.943897 671 hybrid_clock.cc:648] HybridClock initialized: now 1790576574943863 us; error 46 us; skew 500 ppm
Sep 28 06:22:54 dist-test-slave-q03f krb5kdc[658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576574, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:54.946692 671 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:22:54.947148 671 webserver.cc:460] Webserver started at http://127.31.214.126:44193/ using document root <none> and password file <none>
I20260928 06:22:54.947405 671 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:54.947491 671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:54.947641 671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:54.948815 671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "434dfdea531845f1b63e41fd74e8ef7a"
format_stamp: "Formatted at 2026-09-28 06:22:54 on dist-test-slave-q03f"
server_key: "dd3e438df27a14f98b9d58963b167887"
server_key_iv: "c714ec6a8eaa488330d1a51f26c0d7c8"
server_key_version: "encryptionkey@0"
I20260928 06:22:54.949186 671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "434dfdea531845f1b63e41fd74e8ef7a"
format_stamp: "Formatted at 2026-09-28 06:22:54 on dist-test-slave-q03f"
server_key: "dd3e438df27a14f98b9d58963b167887"
server_key_iv: "c714ec6a8eaa488330d1a51f26c0d7c8"
server_key_version: "encryptionkey@0"
I20260928 06:22:54.950646 671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:22:54.951400 686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:54.951707 671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:54.951860 671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "434dfdea531845f1b63e41fd74e8ef7a"
format_stamp: "Formatted at 2026-09-28 06:22:54 on dist-test-slave-q03f"
server_key: "dd3e438df27a14f98b9d58963b167887"
server_key_iv: "c714ec6a8eaa488330d1a51f26c0d7c8"
server_key_version: "encryptionkey@0"
I20260928 06:22:54.951951 671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:54.963861 671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:54.966464 671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:54.966708 671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:54.970850 671 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:36633
I20260928 06:22:54.970886 738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:36633 every 8 connection(s)
I20260928 06:22:54.971252 671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:22:54.972187 739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:54.974674 739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a: Bootstrap starting.
I20260928 06:22:54.975281 739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:54.975605 739 log.cc:824] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:54.975729 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 671
I20260928 06:22:54.975823 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:22:54.975987 32601 external_mini_cluster.cc:1428] Setting key f71469a7d8503ed3a1b772bc113c52ad
I20260928 06:22:54.976449 739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a: No bootstrap required, opened a new log
I20260928 06:22:54.978590 739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434dfdea531845f1b63e41fd74e8ef7a" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36633 } }
I20260928 06:22:54.978811 739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:54.978859 739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 434dfdea531845f1b63e41fd74e8ef7a, State: Initialized, Role: FOLLOWER
I20260928 06:22:54.979014 739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434dfdea531845f1b63e41fd74e8ef7a" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36633 } }
I20260928 06:22:54.979110 739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:22:54.979147 739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:22:54.979182 739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:54.979694 739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434dfdea531845f1b63e41fd74e8ef7a" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36633 } }
I20260928 06:22:54.979779 739 leader_election.cc:302] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 434dfdea531845f1b63e41fd74e8ef7a; no voters:
I20260928 06:22:54.979982 739 leader_election.cc:288] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:22:54.980134 744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:22:54.980274 739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:22:54.980332 744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [term 1 LEADER]: Becoming Leader. State: Replica: 434dfdea531845f1b63e41fd74e8ef7a, State: Running, Role: LEADER
I20260928 06:22:54.980517 744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434dfdea531845f1b63e41fd74e8ef7a" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36633 } }
I20260928 06:22:54.981108 746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "434dfdea531845f1b63e41fd74e8ef7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434dfdea531845f1b63e41fd74e8ef7a" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36633 } } }
I20260928 06:22:54.981216 746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [sys.catalog]: This master's current role is: LEADER
I20260928 06:22:54.981369 746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 434dfdea531845f1b63e41fd74e8ef7a. Latest consensus state: current_term: 1 leader_uuid: "434dfdea531845f1b63e41fd74e8ef7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "434dfdea531845f1b63e41fd74e8ef7a" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36633 } } }
I20260928 06:22:54.981419 746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a [sys.catalog]: This master's current role is: LEADER
I20260928 06:22:54.981442 753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:22:54.981982 753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:22:54.983954 753 catalog_manager.cc:1347] Generated new cluster ID: 2197af0bfe264cb3b78801a38235a31c
I20260928 06:22:54.984009 753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:22:54 dist-test-slave-q03f krb5kdc[658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:54.992089 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:54.976571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.31.214.126:36633)
0928 06:22:54.976779 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:54.976788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:54.976822 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:22:54.977749 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:54.977759 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:54.977826 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:54.978087 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:54.978694 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:54.981576 (+ 2882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:54.982708 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:54.983329 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:54.983453 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:54.988198 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:54.988203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:54.988204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:54.988217 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:54.990351 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:54.990793 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:54.990796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:54.990797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:54.990907 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:54.991128 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:54.991129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:54.991129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:54.991437 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:54.991588 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:54.991727 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 06:22:54.991865 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260928 06:22:55.007143 753 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:22:55.007786 753 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:22:55.011896 753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 434dfdea531845f1b63e41fd74e8ef7a: Generated new TSK 0
I20260928 06:22:55.012166 753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:22:55.050544 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:36633
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:55.135450 765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:55.135715 765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:55.135756 765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:55.136971 765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:55.137040 765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:55.137087 765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:22:55.138814 765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:36633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:55.139087 765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:55.139375 765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:55.144779 770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:55.144760 771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:55.145040 765 server_base.cc:1034] running on GCE node
W20260928 06:22:55.144761 773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:55.145397 765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:55.145697 765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:55.146872 765 hybrid_clock.cc:648] HybridClock initialized: now 1790576575146834 us; error 59 us; skew 500 ppm
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:55.149688 765 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:22:55.150266 765 webserver.cc:460] Webserver started at http://127.31.214.65:43543/ using document root <none> and password file <none>
I20260928 06:22:55.150507 765 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:55.150588 765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:55.150744 765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:55.151878 765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "3b481118196849f1ba8d2996e896edac"
format_stamp: "Formatted at 2026-09-28 06:22:55 on dist-test-slave-q03f"
server_key: "032cc0ddc25ae898afc1c1badf26d766"
server_key_iv: "2e13bcf375ef716dc51ea2dad52aa0f3"
server_key_version: "encryptionkey@0"
I20260928 06:22:55.152274 765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "3b481118196849f1ba8d2996e896edac"
format_stamp: "Formatted at 2026-09-28 06:22:55 on dist-test-slave-q03f"
server_key: "032cc0ddc25ae898afc1c1badf26d766"
server_key_iv: "2e13bcf375ef716dc51ea2dad52aa0f3"
server_key_version: "encryptionkey@0"
I20260928 06:22:55.153756 765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:22:55.154755 780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:55.154969 765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:22:55.155086 765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "3b481118196849f1ba8d2996e896edac"
format_stamp: "Formatted at 2026-09-28 06:22:55 on dist-test-slave-q03f"
server_key: "032cc0ddc25ae898afc1c1badf26d766"
server_key_iv: "2e13bcf375ef716dc51ea2dad52aa0f3"
server_key_version: "encryptionkey@0"
I20260928 06:22:55.155179 765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:55.177314 765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:55.180001 765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:55.180198 765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:55.180492 765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:22:55.180833 765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:55.180892 765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:55.180928 765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:55.180979 765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:55.187700 765 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:39839
I20260928 06:22:55.187742 893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:39839 every 8 connection(s)
I20260928 06:22:55.188102 765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:22:55.196605 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 765
I20260928 06:22:55.196698 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:22:55.196888 32601 external_mini_cluster.cc:1428] Setting key 2906eaf7e870c2b285ebeb90f50cfd4c
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:55.201256 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.188902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:55541 (local address 127.31.214.126:36633)
0928 06:22:55.188977 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.188979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.189174 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:22:55.192520 (+ 3346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.192522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.192535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.192609 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:55.193306 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.193874 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.195079 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.195307 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.195366 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.197800 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:55.197816 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:55.197816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:55.197830 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:55.199680 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.200191 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.200192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.200193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.200253 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.200533 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.200534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.200535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.200667 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:55.200741 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.201040 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.201160 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:22:55.201428 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.188871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:36633 (local address 127.31.214.65:55541)
0928 06:22:55.189036 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:55.189081 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:55.192378 (+ 3297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:55.192706 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:55.192710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:55.193000 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:55.193215 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.193219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.194045 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.194046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.194991 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.194994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.195388 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.195389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.195557 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.196195 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:55.196204 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:55.197699 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:55.199780 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:55.199788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:55.199790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:55.200122 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:55.200335 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:55.200336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:55.200336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:55.200462 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:55.200743 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:55.200751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:55.200900 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 06:22:55.201006 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 06:22:55.201180 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260928 06:22:55.202173 894 heartbeater.cc:347] Connected to a master server at 127.31.214.126:36633
I20260928 06:22:55.202277 894 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:55.202487 894 heartbeater.cc:511] Master 127.31.214.126:36633 requested a full tablet report, sending...
I20260928 06:22:55.203053 703 ts_manager.cc:194] Registered new tserver with Master: 3b481118196849f1ba8d2996e896edac (127.31.214.65:39839)
I20260928 06:22:55.203712 703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:55541
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:22:55.227087 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:36633
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:55.311656 899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:55.311898 899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:55.311926 899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:55.313210 899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:55.313258 899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:55.313287 899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:22:55.314858 899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:36633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:55.315110 899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:55.315387 899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:55.320640 905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:55.320640 907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:55.320747 904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:55.320982 899 server_base.cc:1034] running on GCE node
I20260928 06:22:55.321228 899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:55.321434 899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:55.322595 899 hybrid_clock.cc:648] HybridClock initialized: now 1790576575322574 us; error 38 us; skew 500 ppm
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:55.325265 899 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:22:55.325747 899 webserver.cc:460] Webserver started at http://127.31.214.66:36057/ using document root <none> and password file <none>
I20260928 06:22:55.326030 899 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:55.326105 899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:55.326259 899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:55.327337 899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "9c87bdaa3053408284db0d127a11366c"
format_stamp: "Formatted at 2026-09-28 06:22:55 on dist-test-slave-q03f"
server_key: "a102b3e4fd39b891ba153e92368d3201"
server_key_iv: "f34973bc0029c0cf20cd07b0e8993bdf"
server_key_version: "encryptionkey@0"
I20260928 06:22:55.327731 899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "9c87bdaa3053408284db0d127a11366c"
format_stamp: "Formatted at 2026-09-28 06:22:55 on dist-test-slave-q03f"
server_key: "a102b3e4fd39b891ba153e92368d3201"
server_key_iv: "f34973bc0029c0cf20cd07b0e8993bdf"
server_key_version: "encryptionkey@0"
I20260928 06:22:55.329205 899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:22:55.330089 914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:55.330288 899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:55.330396 899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "9c87bdaa3053408284db0d127a11366c"
format_stamp: "Formatted at 2026-09-28 06:22:55 on dist-test-slave-q03f"
server_key: "a102b3e4fd39b891ba153e92368d3201"
server_key_iv: "f34973bc0029c0cf20cd07b0e8993bdf"
server_key_version: "encryptionkey@0"
I20260928 06:22:55.330479 899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:55.342427 899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:55.344909 899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:55.345086 899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:55.345335 899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:22:55.345659 899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:55.345712 899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:55.345753 899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:55.345777 899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:55.352035 899 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:39581
I20260928 06:22:55.352135 1027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:39581 every 8 connection(s)
I20260928 06:22:55.352473 899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:55.362435 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 899
I20260928 06:22:55.362509 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:22:55.362684 32601 external_mini_cluster.cc:1428] Setting key 8b2899ced71392bb903f14b81ca7182b
I20260928 06:22:55.362896 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.353247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:51841 (local address 127.31.214.126:36633)
0928 06:22:55.353293 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.353296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.353484 (+ 188us) server_negotiation.cc:407] Connection header received
0928 06:22:55.354686 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.354687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.354702 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.354754 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:55.355397 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.355981 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.356998 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.357208 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.357299 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.359597 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:55.359598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:55.359598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:55.359607 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:55.361296 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.361775 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.361776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.361777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.361843 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.362141 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.362142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.362142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.362286 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:55.362358 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.362705 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.362804 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:22:55.363075 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.353208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:36633 (local address 127.31.214.66:51841)
0928 06:22:55.353344 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:55.353395 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:55.354558 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:55.354822 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:55.354826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:55.355094 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:55.355300 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.355304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.356077 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.356078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.356905 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.356907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.357306 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.357307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.357438 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.357916 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:55.357924 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:55.359502 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:55.361404 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:55.361411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:55.361414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:55.361690 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:55.361917 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:55.361917 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:55.361918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:55.362063 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:55.362361 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:55.362368 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:55.362540 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 06:22:55.362669 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 06:22:55.362833 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260928 06:22:55.363507 1028 heartbeater.cc:347] Connected to a master server at 127.31.214.126:36633
I20260928 06:22:55.363638 1028 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:55.363818 1028 heartbeater.cc:511] Master 127.31.214.126:36633 requested a full tablet report, sending...
I20260928 06:22:55.364217 703 ts_manager.cc:194] Registered new tserver with Master: 9c87bdaa3053408284db0d127a11366c (127.31.214.66:39581)
I20260928 06:22:55.364753 703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:51841
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:22:55.389770 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:36633
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:55.473644 1033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:55.473884 1033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:55.473920 1033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:55.475086 1033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:55.475158 1033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:55.475205 1033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:22:55.476771 1033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:36633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:55.477038 1033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:55.477283 1033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:55.482514 1039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:55.482528 1041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:55.482621 1038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:55.482954 1033 server_base.cc:1034] running on GCE node
I20260928 06:22:55.483201 1033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:55.483443 1033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:55.484627 1033 hybrid_clock.cc:648] HybridClock initialized: now 1790576575484601 us; error 34 us; skew 500 ppm
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:55.487376 1033 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:22:55.487879 1033 webserver.cc:460] Webserver started at http://127.31.214.67:32849/ using document root <none> and password file <none>
I20260928 06:22:55.488114 1033 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:55.488183 1033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:55.488336 1033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:55.489424 1033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "6b360e2df1374a8c8d17dbed9210abd8"
format_stamp: "Formatted at 2026-09-28 06:22:55 on dist-test-slave-q03f"
server_key: "5d6403eadaa427f48595d9e0c4733849"
server_key_iv: "f91b01c5b624904cf43cfe0697b37017"
server_key_version: "encryptionkey@0"
I20260928 06:22:55.489794 1033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "6b360e2df1374a8c8d17dbed9210abd8"
format_stamp: "Formatted at 2026-09-28 06:22:55 on dist-test-slave-q03f"
server_key: "5d6403eadaa427f48595d9e0c4733849"
server_key_iv: "f91b01c5b624904cf43cfe0697b37017"
server_key_version: "encryptionkey@0"
I20260928 06:22:55.491194 1033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:22:55.491941 1048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:55.492137 1033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:22:55.492237 1033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "6b360e2df1374a8c8d17dbed9210abd8"
format_stamp: "Formatted at 2026-09-28 06:22:55 on dist-test-slave-q03f"
server_key: "5d6403eadaa427f48595d9e0c4733849"
server_key_iv: "f91b01c5b624904cf43cfe0697b37017"
server_key_version: "encryptionkey@0"
I20260928 06:22:55.492323 1033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:55.502466 1033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:55.504904 1033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:55.505082 1033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:55.505339 1033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:22:55.505709 1033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:55.505765 1033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:55.505806 1033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:55.505834 1033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:55.512674 1033 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:39159
I20260928 06:22:55.512759 1161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:39159 every 8 connection(s)
I20260928 06:22:55.513114 1033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:22:55.514984 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 1033
I20260928 06:22:55.515059 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:22:55.515234 32601 external_mini_cluster.cc:1428] Setting key 774e29c0f08e0ddeafbff3caee591263
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:55.523650 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.513892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:41127 (local address 127.31.214.126:36633)
0928 06:22:55.513974 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.513977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.514241 (+ 264us) server_negotiation.cc:407] Connection header received
0928 06:22:55.515305 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.515307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.515321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.515372 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:55.516207 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.516727 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.517789 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.518083 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.518169 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.520636 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:55.520638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:55.520638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:55.520646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:55.522182 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.522629 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.522630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.522630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.522693 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.522943 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.522944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.522945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.523067 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:55.523148 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.523465 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.523547 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:22:55.523903 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.513890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:36633 (local address 127.31.214.67:41127)
0928 06:22:55.514112 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:55.514151 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:55.515181 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:55.515444 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:55.515448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:55.515775 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:55.516114 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.516120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.516823 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.516824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.517694 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.517697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.518179 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.518180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.518316 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.518952 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:55.518962 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:55.520527 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:55.522298 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:55.522304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:55.522307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:55.522536 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:55.522766 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:55.522767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:55.522767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:55.522876 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:55.523138 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:55.523145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:55.523322 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 06:22:55.523431 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:22:55.523596 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":108,"threads_started":1}
I20260928 06:22:55.524402 1162 heartbeater.cc:347] Connected to a master server at 127.31.214.126:36633
I20260928 06:22:55.524487 1162 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:55.524735 1162 heartbeater.cc:511] Master 127.31.214.126:36633 requested a full tablet report, sending...
I20260928 06:22:55.525198 703 ts_manager.cc:194] Registered new tserver with Master: 6b360e2df1374a8c8d17dbed9210abd8 (127.31.214.67:39159)
I20260928 06:22:55.525640 703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:41127
I20260928 06:22:55.526901 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:22:55.535364 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.528446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.31.214.126:36633)
0928 06:22:55.528501 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.528503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.528628 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:22:55.528666 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.528667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.528675 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.528723 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:55.529249 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.529748 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.530661 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.530844 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.530901 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.532267 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:55.532268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:55.532268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:55.532274 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:55.533907 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.534346 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.534347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.534347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.534400 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.534794 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.534795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.534795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.534969 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:55.535070 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.535215 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.535288 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:22:55.537915 703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:22:55.539937 703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:22:55.547242 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.543245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44282 (local address 127.31.214.67:39159)
0928 06:22:55.543814 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.543819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.543830 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:22:55.543893 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.543895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.543911 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.543992 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:55.544322 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.544999 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.546325 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.546822 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.546913 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.546977 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.547112 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.547156 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":145,"threads_started":1}
I20260928 06:22:55.547233 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.543003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:39159 (local address 127.0.0.1:44282)
0928 06:22:55.543415 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:55.543457 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:55.543545 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:55.544186 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:55.544187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:55.544188 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:55.544260 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.544262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.545114 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.545115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.546228 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.546231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.546912 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.546913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.546992 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.547006 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:22:55.547098 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:22:55.547157 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":73,"threads_started":1}
I20260928 06:22:55.548136 1096 tablet_service.cc:1467] Processing CreateTablet for tablet 064044b506004aceb94e8c3198257154 (DEFAULT_TABLE table=test-table [id=716aaafb949148cb82abb66135bdca20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:22:55.548477 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.543255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:39581 (local address 127.0.0.1:47710)
0928 06:22:55.543651 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:55.543660 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:55.543697 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:55.544135 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:55.544136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:55.544138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:55.544189 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.544191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.545697 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.545698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.547049 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.547053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.548193 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.548194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.548276 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.548282 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:22:55.548343 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:22:55.548387 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260928 06:22:55.548556 1096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064044b506004aceb94e8c3198257154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:55.548784 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.543003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:39839 (local address 127.0.0.1:48976)
0928 06:22:55.543558 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:55.543569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:55.543615 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:55.543837 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:55.543841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:55.543844 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:55.543929 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.543932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.545729 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.545729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.547057 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.547059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.548443 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.548444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.548595 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.548601 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:22:55.548666 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:22:55.548713 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":70,"thread_start_us":75,"threads_started":1}
I20260928 06:22:55.548933 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.543367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.31.214.66:39581)
0928 06:22:55.543599 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.543604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.543943 (+ 339us) server_negotiation.cc:407] Connection header received
0928 06:22:55.543992 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.543995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.544012 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.544126 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:55.544278 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.545476 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.547168 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.547981 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.548089 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.548150 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.548711 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.548758 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
I20260928 06:22:55.549132 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.543136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.31.214.65:39839)
0928 06:22:55.543450 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.543457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.543636 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:22:55.543705 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.543708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.543728 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.543823 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:55.544005 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.545474 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.547169 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.548277 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.548351 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.548407 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.548903 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.548961 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":96,"threads_started":1}
I20260928 06:22:55.550007 828 tablet_service.cc:1467] Processing CreateTablet for tablet 064044b506004aceb94e8c3198257154 (DEFAULT_TABLE table=test-table [id=716aaafb949148cb82abb66135bdca20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:22:55.550436 828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064044b506004aceb94e8c3198257154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:55.550578 962 tablet_service.cc:1467] Processing CreateTablet for tablet 064044b506004aceb94e8c3198257154 (DEFAULT_TABLE table=test-table [id=716aaafb949148cb82abb66135bdca20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:22:55.550966 962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064044b506004aceb94e8c3198257154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:55.552742 1179 tablet_bootstrap.cc:492] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8: Bootstrap starting.
I20260928 06:22:55.553580 1179 tablet_bootstrap.cc:654] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:55.554268 1179 log.cc:824] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:55.554836 1181 tablet_bootstrap.cc:492] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac: Bootstrap starting.
I20260928 06:22:55.555166 1182 tablet_bootstrap.cc:492] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c: Bootstrap starting.
I20260928 06:22:55.555434 1179 tablet_bootstrap.cc:492] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8: No bootstrap required, opened a new log
I20260928 06:22:55.555565 1179 ts_tablet_manager.cc:1397] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:22:55.555684 1181 tablet_bootstrap.cc:654] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:55.555930 1182 tablet_bootstrap.cc:654] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:55.556414 1181 log.cc:824] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:55.556546 1182 log.cc:824] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:55.557216 1181 tablet_bootstrap.cc:492] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac: No bootstrap required, opened a new log
I20260928 06:22:55.557294 1181 ts_tablet_manager.cc:1397] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:22:55.557518 1182 tablet_bootstrap.cc:492] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c: No bootstrap required, opened a new log
I20260928 06:22:55.557590 1182 ts_tablet_manager.cc:1397] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:22:55.557857 1179 raft_consensus.cc:348] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } }
I20260928 06:22:55.558135 1179 raft_consensus.cc:374] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:55.558192 1179 raft_consensus.cc:729] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b360e2df1374a8c8d17dbed9210abd8, State: Initialized, Role: FOLLOWER
I20260928 06:22:55.558365 1179 consensus_queue.cc:260] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } }
I20260928 06:22:55.558698 1179 ts_tablet_manager.cc:1428] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:22:55.558866 1162 heartbeater.cc:503] Master 127.31.214.126:36633 was elected leader, sending a full tablet report...
I20260928 06:22:55.559458 1181 raft_consensus.cc:348] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } }
I20260928 06:22:55.559652 1181 raft_consensus.cc:374] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:55.559687 1181 raft_consensus.cc:729] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b481118196849f1ba8d2996e896edac, State: Initialized, Role: FOLLOWER
I20260928 06:22:55.559810 1181 consensus_queue.cc:260] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } }
I20260928 06:22:55.559830 1182 raft_consensus.cc:348] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } }
I20260928 06:22:55.560006 1182 raft_consensus.cc:374] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:55.560055 1182 raft_consensus.cc:729] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c87bdaa3053408284db0d127a11366c, State: Initialized, Role: FOLLOWER
I20260928 06:22:55.560190 1182 consensus_queue.cc:260] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } }
I20260928 06:22:55.560279 894 heartbeater.cc:503] Master 127.31.214.126:36633 was elected leader, sending a full tablet report...
I20260928 06:22:55.560202 1181 ts_tablet_manager.cc:1428] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:22:55.560632 1182 ts_tablet_manager.cc:1428] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:22:55.560786 1028 heartbeater.cc:503] Master 127.31.214.126:36633 was elected leader, sending a full tablet report...
W20260928 06:22:55.603116 1029 tablet.cc:2369] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:22:55.688868 895 tablet.cc:2369] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:22:55.763729 1163 tablet.cc:2369] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:22:55.872634 1185 raft_consensus.cc:482] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:22:55.872777 1185 raft_consensus.cc:504] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } }
I20260928 06:22:55.873217 1185 leader_election.cc:288] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b481118196849f1ba8d2996e896edac (127.31.214.65:39839), 9c87bdaa3053408284db0d127a11366c (127.31.214.66:39581)
I20260928 06:22:55.876659 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.873568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:57253 (local address 127.31.214.65:39839)
0928 06:22:55.873660 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.873663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.873676 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:22:55.873712 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.873713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.873727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.873811 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:55.874033 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.874707 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.875760 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.876251 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.876309 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.876318 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.876510 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.876575 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:22:55.876658 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.873465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:39839 (local address 127.31.214.67:57253)
0928 06:22:55.873529 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:55.873542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:55.873643 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:55.873853 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:55.873854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:55.873856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:55.873918 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.873920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.874800 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.874801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.875655 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.875658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.876333 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.876334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.876418 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.876422 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:22:55.876484 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:22:55.876535 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":5}
I20260928 06:22:55.876874 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.873640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:39581 (local address 127.31.214.67:37071)
0928 06:22:55.873850 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:55.873863 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:55.873986 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:55.874174 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:55.874175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:55.874177 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:55.874255 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.874257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.874956 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.874957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.875937 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:55.875940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.876569 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.876570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:55.876648 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.876653 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:22:55.876717 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:22:55.876761 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":124,"threads_started":1}
I20260928 06:22:55.876879 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.873737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:37071 (local address 127.31.214.66:39581)
0928 06:22:55.873813 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.873817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.873972 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:22:55.874050 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.874051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.874062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.874142 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:55.874337 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.874839 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.876047 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.876484 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.876546 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.876553 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.876742 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.876788 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:22:55.877316 848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064044b506004aceb94e8c3198257154" candidate_uuid: "6b360e2df1374a8c8d17dbed9210abd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b481118196849f1ba8d2996e896edac" is_pre_election: true
I20260928 06:22:55.877566 848 raft_consensus.cc:2454] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b360e2df1374a8c8d17dbed9210abd8 in term 0.
I20260928 06:22:55.877436 982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064044b506004aceb94e8c3198257154" candidate_uuid: "6b360e2df1374a8c8d17dbed9210abd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c87bdaa3053408284db0d127a11366c" is_pre_election: true
I20260928 06:22:55.877718 982 raft_consensus.cc:2454] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b360e2df1374a8c8d17dbed9210abd8 in term 0.
I20260928 06:22:55.877839 1049 leader_election.cc:302] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b481118196849f1ba8d2996e896edac, 6b360e2df1374a8c8d17dbed9210abd8; no voters:
I20260928 06:22:55.878000 1185 raft_consensus.cc:2781] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:22:55.878062 1185 raft_consensus.cc:482] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:22:55.878089 1185 raft_consensus.cc:3037] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:55.878820 1185 raft_consensus.cc:504] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } }
I20260928 06:22:55.878962 1185 leader_election.cc:288] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [CANDIDATE]: Term 1 election: Requested vote from peers 3b481118196849f1ba8d2996e896edac (127.31.214.65:39839), 9c87bdaa3053408284db0d127a11366c (127.31.214.66:39581)
I20260928 06:22:55.879171 848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064044b506004aceb94e8c3198257154" candidate_uuid: "6b360e2df1374a8c8d17dbed9210abd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b481118196849f1ba8d2996e896edac"
I20260928 06:22:55.879179 982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064044b506004aceb94e8c3198257154" candidate_uuid: "6b360e2df1374a8c8d17dbed9210abd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c87bdaa3053408284db0d127a11366c"
I20260928 06:22:55.879245 848 raft_consensus.cc:3037] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:55.879297 982 raft_consensus.cc:3037] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:55.880097 848 raft_consensus.cc:2454] T 064044b506004aceb94e8c3198257154 P 3b481118196849f1ba8d2996e896edac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b360e2df1374a8c8d17dbed9210abd8 in term 1.
I20260928 06:22:55.880275 1049 leader_election.cc:302] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b481118196849f1ba8d2996e896edac, 6b360e2df1374a8c8d17dbed9210abd8; no voters:
I20260928 06:22:55.880340 982 raft_consensus.cc:2454] T 064044b506004aceb94e8c3198257154 P 9c87bdaa3053408284db0d127a11366c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b360e2df1374a8c8d17dbed9210abd8 in term 1.
I20260928 06:22:55.880446 1185 raft_consensus.cc:2781] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:22:55.880676 1185 raft_consensus.cc:686] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [term 1 LEADER]: Becoming Leader. State: Replica: 6b360e2df1374a8c8d17dbed9210abd8, State: Running, Role: LEADER
I20260928 06:22:55.880808 1185 consensus_queue.cc:237] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } }
I20260928 06:22:55.881911 703 catalog_manager.cc:5495] T 064044b506004aceb94e8c3198257154 P 6b360e2df1374a8c8d17dbed9210abd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b360e2df1374a8c8d17dbed9210abd8 (127.31.214.67). New cstate: current_term: 1 leader_uuid: "6b360e2df1374a8c8d17dbed9210abd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b481118196849f1ba8d2996e896edac" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 39839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b360e2df1374a8c8d17dbed9210abd8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c87bdaa3053408284db0d127a11366c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 39581 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:22:55.913122 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.904876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.31.214.65:39839)
0928 06:22:55.904997 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.904999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.905067 (+ 68us) server_negotiation.cc:407] Connection header received
0928 06:22:55.905117 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.905118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.905138 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.905203 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:55.905798 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.906404 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.907246 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.907439 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.907515 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.910075 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:55.910087 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:55.910088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:55.910095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:55.911839 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.912198 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.912202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.912202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.912286 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.912502 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.912503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.912504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.912694 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:55.912819 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.912887 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.912966 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:22:55.940302 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.931779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.31.214.65:39839)
0928 06:22:55.931850 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.931852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.931862 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:22:55.931897 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.931899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.931908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.931976 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:55.932621 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.933105 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.933918 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.934173 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.934252 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.936902 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:55.936904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:55.936904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:55.936915 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:55.938660 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.939114 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.939115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.939115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.939207 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.939538 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.939538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.939538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.939682 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:55.939759 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.940053 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.940185 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:22:55 dist-test-slave-q03f krb5kdc[658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:22:55.967664 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:55.958294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.31.214.65:39839)
0928 06:22:55.958356 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:55.958358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:55.958576 (+ 218us) server_negotiation.cc:407] Connection header received
0928 06:22:55.958611 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:55.958612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:55.958625 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:55.958696 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:55.959426 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.960043 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.960957 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:55.961174 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:55.961260 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:55.964431 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:55.964432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:55.964432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:55.964442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:55.966236 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.966698 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.966699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.966699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.966767 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:55.967086 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:55.967087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:55.967088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:55.967216 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:55.967306 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:55.967490 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 06:22:55.967559 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 06:22:55.968237 808 tablet_service.cc:2908] Rejecting scan request for tablet 064044b506004aceb94e8c3198257154: Uninitialized: clean time has not yet been initialized
I20260928 06:22:55.969385 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 765
I20260928 06:22:55.976063 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 899
I20260928 06:22:55.981628 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 1033
I20260928 06:22:55.986732 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 671
2026-09-28T06:22:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2370 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:22:56 dist-test-slave-q03f krb5kdc[1203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:22:56 dist-test-slave-q03f krb5kdc[1203](info): set up 2 sockets
Sep 28 06:22:56 dist-test-slave-q03f krb5kdc[1203](info): commencing operation
krb5kdc: starting...
W20260928 06:22:57.125727 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:22:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:22:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:22:57.254014 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:44099
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:44099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:57.338438 1216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:57.338657 1216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:57.338681 1216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:57.339799 1216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:22:57.339838 1216 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:22:57.339851 1216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:57.339864 1216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:57.339877 1216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:22:57.339891 1216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:22:57.341356 1216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:44099
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:57.341576 1216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:57.341758 1216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:57.347025 1224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:57.347029 1221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:57.347128 1222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:57.347525 1216 server_base.cc:1034] running on GCE node
I20260928 06:22:57.347772 1216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:57.348080 1216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:57.349253 1216 hybrid_clock.cc:648] HybridClock initialized: now 1790576577349240 us; error 33 us; skew 500 ppm
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:57.359185 1216 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:22:57.359879 1216 webserver.cc:460] Webserver started at http://127.31.214.126:37049/ using document root <none> and password file <none>
I20260928 06:22:57.360162 1216 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:57.360236 1216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:57.360405 1216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:57.361591 1216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "74c0c2cd271e419dbc85856a61fe1fcf"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "9a73ec929bb47ebc74997cb4f8dd1c6f"
server_key_iv: "ff05ff5ce90f49dedc783df92670af1f"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.362067 1216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "74c0c2cd271e419dbc85856a61fe1fcf"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "9a73ec929bb47ebc74997cb4f8dd1c6f"
server_key_iv: "ff05ff5ce90f49dedc783df92670af1f"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.363684 1216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:22:57.364596 1231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:57.364862 1216 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:57.365007 1216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "74c0c2cd271e419dbc85856a61fe1fcf"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "9a73ec929bb47ebc74997cb4f8dd1c6f"
server_key_iv: "ff05ff5ce90f49dedc783df92670af1f"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.365095 1216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:57.385469 1216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:57.388285 1216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:57.388487 1216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:57.394114 1216 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:44099
I20260928 06:22:57.394171 1293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:44099 every 8 connection(s)
I20260928 06:22:57.394550 1216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:22:57.395277 1294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:57.397758 1294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf: Bootstrap starting.
I20260928 06:22:57.398450 1294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:57.398725 1294 log.cc:824] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:57.399526 1294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf: No bootstrap required, opened a new log
I20260928 06:22:57.399813 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 1216
I20260928 06:22:57.399916 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:22:57.400108 32601 external_mini_cluster.cc:1428] Setting key b059c6b8b19e54965eb3569ed2f73645
I20260928 06:22:57.402690 1294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c0c2cd271e419dbc85856a61fe1fcf" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44099 } }
I20260928 06:22:57.402896 1294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:57.402942 1294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c0c2cd271e419dbc85856a61fe1fcf, State: Initialized, Role: FOLLOWER
I20260928 06:22:57.403110 1294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c0c2cd271e419dbc85856a61fe1fcf" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44099 } }
I20260928 06:22:57.403182 1294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:22:57.403206 1294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:22:57.403276 1294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:57.403952 1294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c0c2cd271e419dbc85856a61fe1fcf" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44099 } }
I20260928 06:22:57.404069 1294 leader_election.cc:302] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74c0c2cd271e419dbc85856a61fe1fcf; no voters:
I20260928 06:22:57.404251 1294 leader_election.cc:288] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:22:57.404376 1299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:22:57.404532 1294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:22:57.404587 1299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [term 1 LEADER]: Becoming Leader. State: Replica: 74c0c2cd271e419dbc85856a61fe1fcf, State: Running, Role: LEADER
I20260928 06:22:57.404752 1299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c0c2cd271e419dbc85856a61fe1fcf" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44099 } }
I20260928 06:22:57.405175 1301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c0c2cd271e419dbc85856a61fe1fcf. Latest consensus state: current_term: 1 leader_uuid: "74c0c2cd271e419dbc85856a61fe1fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c0c2cd271e419dbc85856a61fe1fcf" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44099 } } }
I20260928 06:22:57.405341 1301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [sys.catalog]: This master's current role is: LEADER
I20260928 06:22:57.405618 1300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74c0c2cd271e419dbc85856a61fe1fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c0c2cd271e419dbc85856a61fe1fcf" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44099 } } }
I20260928 06:22:57.405706 1300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf [sys.catalog]: This master's current role is: LEADER
I20260928 06:22:57.405843 1303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:22:57.406458 1303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:22:57.408602 1303 catalog_manager.cc:1347] Generated new cluster ID: 2d041806ce844f6ea2f0c83016d29c72
I20260928 06:22:57.408674 1303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:57.415241 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.400695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.31.214.126:44099)
0928 06:22:57.400924 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.400935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.400974 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:22:57.402634 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.402645 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.402717 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.402885 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:57.403760 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.405284 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.406680 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.407493 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.407614 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.410618 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:57.410622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:57.410622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:57.410636 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:57.413377 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.413827 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.413830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.413831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.413971 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.414192 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.414193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.414193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.414547 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:57.414714 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.414852 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.415007 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
I20260928 06:22:57.416047 1303 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:22:57.416678 1303 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:22:57.423578 1303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf: Generated new TSK 0
I20260928 06:22:57.423812 1303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:22:57.457911 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:44099
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:57.543886 1320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:57.544097 1320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:57.544122 1320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:57.545235 1320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:57.545274 1320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:57.545295 1320 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:22:57.545311 1320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:22:57.546727 1320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.214.126:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:57.546923 1320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:57.547122 1320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:57.552443 1325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:57.552443 1328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:57.552678 1326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:57.552683 1320 server_base.cc:1034] running on GCE node
I20260928 06:22:57.552961 1320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:57.553207 1320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:57.554430 1320 hybrid_clock.cc:648] HybridClock initialized: now 1790576577554382 us; error 63 us; skew 500 ppm
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:57.557278 1320 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:22:57.557823 1320 webserver.cc:460] Webserver started at http://127.31.214.65:33399/ using document root <none> and password file <none>
I20260928 06:22:57.558121 1320 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:57.558193 1320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:57.558355 1320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:57.559463 1320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "088536e2db5e4931bf5bcf0a34245284"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "218d9b2774a68d3670fa79f5e31088ca"
server_key_iv: "b83b33c7be4f154ab656e7b8d951e60f"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.559842 1320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "088536e2db5e4931bf5bcf0a34245284"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "218d9b2774a68d3670fa79f5e31088ca"
server_key_iv: "b83b33c7be4f154ab656e7b8d951e60f"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.561311 1320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:22:57.562204 1335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:57.562402 1320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:57.562601 1320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "088536e2db5e4931bf5bcf0a34245284"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "218d9b2774a68d3670fa79f5e31088ca"
server_key_iv: "b83b33c7be4f154ab656e7b8d951e60f"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.562688 1320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:57.583856 1320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:57.586498 1320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:57.586683 1320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:57.587312 1320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:57.587375 1320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:57.587410 1320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:57.587481 1320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:57.595206 1320 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:41441
I20260928 06:22:57.595283 1450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:41441 every 8 connection(s)
I20260928 06:22:57.595696 1320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:22:57.597590 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.587801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:50973 (local address 127.31.214.126:44099)
0928 06:22:57.587873 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.587875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.588100 (+ 225us) server_negotiation.cc:407] Connection header received
0928 06:22:57.588720 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.588722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.588736 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.588796 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:57.589509 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.590144 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.591191 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.591421 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.591501 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.594595 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:57.594597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:57.594597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:57.594606 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:57.596342 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.596824 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.596825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.596825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.596898 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.597101 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.597101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.597101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.597209 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:57.597259 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.597458 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.597524 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:22:57.597859 1344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.587784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44099 (local address 127.31.214.65:50973)
0928 06:22:57.587960 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:57.588002 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:57.588552 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:57.588887 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:57.588891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:57.589196 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:57.589402 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.589407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.590241 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.590242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.591084 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.591086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.591509 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.591510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.591677 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.592296 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:57.592305 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:57.594487 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:57.596436 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.596442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.596444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.596697 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.596949 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.596950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.596950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.597048 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.597248 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:57.597251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:57.597369 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 06:22:57.597447 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:22:57.597574 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260928 06:22:57.598428 1451 heartbeater.cc:347] Connected to a master server at 127.31.214.126:44099
I20260928 06:22:57.598516 1451 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:57.598729 1451 heartbeater.cc:511] Master 127.31.214.126:44099 requested a full tablet report, sending...
I20260928 06:22:57.599366 1247 ts_manager.cc:194] Registered new tserver with Master: 088536e2db5e4931bf5bcf0a34245284 (127.31.214.65:41441)
I20260928 06:22:57.600114 1247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:50973
I20260928 06:22:57.603436 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 1320
I20260928 06:22:57.603530 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:22:57.603662 32601 external_mini_cluster.cc:1428] Setting key 0ba7b10d5e8ca71c5ad053dfc93aa2e0
I20260928 06:22:57.606088 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.599673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44099 (local address 127.31.214.65:51835)
0928 06:22:57.599799 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:57.599810 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:57.599859 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:57.600413 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:57.600414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:57.600640 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:57.600710 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.600713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.601431 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.601432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.602026 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.602029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.602397 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.602398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.602456 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.602901 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:57.602905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:57.603211 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:57.604874 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.604875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.604876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.605117 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.605390 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.605391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.605391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.605462 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.605713 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:57.605714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:57.605764 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:22:57.605850 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:22:57.605983 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260928 06:22:57.606076 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.599738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:51835 (local address 127.31.214.126:44099)
0928 06:22:57.600302 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.600304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.600313 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:22:57.600332 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.600333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.600343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.600396 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:57.600793 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.601338 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.602146 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.602333 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.602388 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.603305 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:57.603306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:57.603307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:57.603313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:57.604794 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.605191 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.605191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.605192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.605243 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.605526 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.605526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.605527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.605628 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:57.605705 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.605878 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.605979 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544}
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:22:57.632642 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:44099
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:57.719520 1460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:57.719765 1460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:57.719794 1460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:57.720978 1460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:57.721030 1460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:57.721055 1460 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:22:57.721073 1460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:22:57.722537 1460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.214.126:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:57.722782 1460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:57.722977 1460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:57.728355 1466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:57.728412 1465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:57.728593 1468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:57.729055 1460 server_base.cc:1034] running on GCE node
I20260928 06:22:57.729300 1460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:57.729555 1460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:57.730702 1460 hybrid_clock.cc:648] HybridClock initialized: now 1790576577730687 us; error 28 us; skew 500 ppm
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:57.733352 1460 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:22:57.733839 1460 webserver.cc:460] Webserver started at http://127.31.214.66:44599/ using document root <none> and password file <none>
I20260928 06:22:57.734109 1460 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:57.734157 1460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:57.734313 1460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:57.735397 1460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "3be5f8b6321c4e938c55591561f2510a"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "a50c495ef11da02da28dd94fb2f22ed7"
server_key_iv: "16fae2509eb00ebb4714a4c4a6dbc401"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.735757 1460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "3be5f8b6321c4e938c55591561f2510a"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "a50c495ef11da02da28dd94fb2f22ed7"
server_key_iv: "16fae2509eb00ebb4714a4c4a6dbc401"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.737180 1460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:22:57.738068 1475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:57.738260 1460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:57.738364 1460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "3be5f8b6321c4e938c55591561f2510a"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "a50c495ef11da02da28dd94fb2f22ed7"
server_key_iv: "16fae2509eb00ebb4714a4c4a6dbc401"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.738454 1460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:57.755622 1460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:57.758245 1460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:57.758414 1460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:57.758967 1460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:57.759032 1460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:57.759121 1460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:57.759155 1460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:57.766125 1460 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:33717
I20260928 06:22:57.766249 1590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:33717 every 8 connection(s)
I20260928 06:22:57.766685 1460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:57.768298 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 1460
I20260928 06:22:57.768404 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:22:57.768600 32601 external_mini_cluster.cc:1428] Setting key 8f266374db378a0788a7f36598d804fd
I20260928 06:22:57.771158 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.759589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:35973 (local address 127.31.214.126:44099)
0928 06:22:57.759666 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.759668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.759879 (+ 211us) server_negotiation.cc:407] Connection header received
0928 06:22:57.760376 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.760377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.760394 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.760469 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:57.761426 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.762405 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.764103 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.764445 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.764531 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.767623 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:57.767625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:57.767625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:57.767634 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:57.769516 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.770020 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.770021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.770022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.770093 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.770337 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.770339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.770339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.770501 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:57.770581 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.770955 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.771059 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:22:57.771577 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.759553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44099 (local address 127.31.214.66:35973)
0928 06:22:57.759740 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:57.759781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:57.760270 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:57.760622 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:57.760626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:57.760991 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:57.761312 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.761318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.762541 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.762542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.763997 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.764000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.764583 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.764584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.764734 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.765423 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:57.765446 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:57.767501 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:57.769625 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.769635 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.769637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.769890 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.770158 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.770159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.770159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.770270 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.770604 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:57.770611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:57.770757 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 06:22:57.770919 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 06:22:57.771170 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":115,"threads_started":1}
I20260928 06:22:57.772207 1591 heartbeater.cc:347] Connected to a master server at 127.31.214.126:44099
I20260928 06:22:57.772301 1591 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:57.772532 1591 heartbeater.cc:511] Master 127.31.214.126:44099 requested a full tablet report, sending...
I20260928 06:22:57.773397 1248 ts_manager.cc:194] Registered new tserver with Master: 3be5f8b6321c4e938c55591561f2510a (127.31.214.66:33717)
I20260928 06:22:57.774026 1248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:35973
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
I20260928 06:22:57.780246 1598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.773399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44099 (local address 127.31.214.66:38041)
0928 06:22:57.773600 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:57.773611 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:57.773676 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:57.773823 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:57.773824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:57.774079 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:57.774247 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.774251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.775262 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.775263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.775795 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.775797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.776280 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.776281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.776338 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.776850 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:57.776854 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:57.777184 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:57.778916 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.778917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.778917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.779083 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.779347 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.779349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.779349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.779441 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.779970 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:57.779971 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:57.780024 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:22:57.780084 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:22:57.780170 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":144,"threads_started":1}
I20260928 06:22:57.780313 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.773432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:38041 (local address 127.31.214.126:44099)
0928 06:22:57.773479 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.773481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.773688 (+ 207us) server_negotiation.cc:407] Connection header received
0928 06:22:57.773735 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.773737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.773750 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.773808 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:57.774331 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.775156 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.775886 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.776199 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.776277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.777304 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:57.777305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:57.777306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:57.777315 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:57.778820 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.779191 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.779192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.779193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.779253 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.779721 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.779722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.779723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.779891 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:57.779967 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.780136 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.780206 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:22:57.798193 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:44099
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:57.883406 1600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:57.883631 1600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:57.883656 1600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:57.884757 1600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:57.884794 1600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:57.884814 1600 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:22:57.884831 1600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:22:57.886299 1600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.214.126:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:57.886504 1600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:57.886689 1600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:57.891808 1606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:57.891917 1600 server_base.cc:1034] running on GCE node
W20260928 06:22:57.891819 1608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:57.891813 1605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:57.892225 1600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:57.892426 1600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:57.893594 1600 hybrid_clock.cc:648] HybridClock initialized: now 1790576577893575 us; error 56 us; skew 500 ppm
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:57.896250 1600 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:22:57.896713 1600 webserver.cc:460] Webserver started at http://127.31.214.67:37909/ using document root <none> and password file <none>
I20260928 06:22:57.896950 1600 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:57.897020 1600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:57.897173 1600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:57.898397 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "f3d171167e3749b7ae35403dfa36dd0e"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "28c5a6ea698039504ab2bcd9ce29bad7"
server_key_iv: "622e135d4fe48eb2561a8b0f5cc1d864"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.898779 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "f3d171167e3749b7ae35403dfa36dd0e"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "28c5a6ea698039504ab2bcd9ce29bad7"
server_key_iv: "622e135d4fe48eb2561a8b0f5cc1d864"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.900264 1600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:22:57.901127 1615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:57.901321 1600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:57.901424 1600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "f3d171167e3749b7ae35403dfa36dd0e"
format_stamp: "Formatted at 2026-09-28 06:22:57 on dist-test-slave-q03f"
server_key: "28c5a6ea698039504ab2bcd9ce29bad7"
server_key_iv: "622e135d4fe48eb2561a8b0f5cc1d864"
server_key_version: "encryptionkey@0"
I20260928 06:22:57.901497 1600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:57.909123 1600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:57.911448 1600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:57.911666 1600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:57.912254 1600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:57.912314 1600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:57.912357 1600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:57.912386 1600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:57.920150 1600 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:34243
I20260928 06:22:57.920235 1730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:34243 every 8 connection(s)
I20260928 06:22:57.920571 1600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:22:57.922680 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.912722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:51619 (local address 127.31.214.126:44099)
0928 06:22:57.912792 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.912794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.913047 (+ 253us) server_negotiation.cc:407] Connection header received
0928 06:22:57.913573 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.913574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.913589 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.913678 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:57.914533 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.915106 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.916419 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.916758 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.916854 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.919601 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:57.919603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:57.919603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:57.919612 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:57.921353 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.921742 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.921743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.921743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.921796 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.922076 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.922077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.922078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.922211 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:57.922279 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.922545 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.922615 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:22:57.922931 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.912711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44099 (local address 127.31.214.67:51619)
0928 06:22:57.912898 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:57.912942 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:57.913467 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:57.913791 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:57.913795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:57.914164 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:57.914422 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.914429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.915272 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.915274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.916311 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.916314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.916865 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.916867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.917033 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.917626 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:57.917666 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:57.919485 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:57.921427 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.921431 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.921433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.921670 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.921858 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.921859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.921859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.922004 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.922286 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:57.922290 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:57.922412 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:22:57.922498 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:22:57.922685 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":118,"threads_started":1}
I20260928 06:22:57.923511 1731 heartbeater.cc:347] Connected to a master server at 127.31.214.126:44099
I20260928 06:22:57.923610 1731 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:57.923720 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 1600
I20260928 06:22:57.923784 1731 heartbeater.cc:511] Master 127.31.214.126:44099 requested a full tablet report, sending...
I20260928 06:22:57.923794 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:22:57.923983 32601 external_mini_cluster.cc:1428] Setting key 02ef8cc043aa137a609896f3e40390fd
I20260928 06:22:57.924254 1248 ts_manager.cc:194] Registered new tserver with Master: f3d171167e3749b7ae35403dfa36dd0e (127.31.214.67:34243)
I20260928 06:22:57.924747 1248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:51619
I20260928 06:22:57.925441 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:22:57.932272 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.925300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:44747 (local address 127.31.214.126:44099)
0928 06:22:57.925338 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.925340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.925348 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:22:57.925368 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.925368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.925378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.925442 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:57.926484 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.926941 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.927919 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.928140 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.928234 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.929316 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:57.929317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:57.929318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:57.929325 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:57.930991 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.931390 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.931391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.931392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.931458 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.931685 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.931686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.931687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.931812 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:57.931897 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.932088 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.932167 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:22:57.932286 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.925102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44099 (local address 127.31.214.67:44747)
0928 06:22:57.925248 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:57.925262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:57.925318 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:57.926025 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:57.926026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:57.926271 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:57.926352 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.926354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.927045 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.927046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.927822 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.927825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.928235 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.928236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.928309 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.928815 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:57.928820 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:57.929187 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:57.931114 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.931116 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.931116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.931308 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.931532 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:57.931533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:57.931533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:57.931612 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:57.931887 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:57.931889 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:57.931951 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 06:22:57.932048 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:22:57.932169 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:22:57 dist-test-slave-q03f krb5kdc[1203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:57.951668 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.943150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.31.214.126:44099)
0928 06:22:57.943225 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.943227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.943237 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:22:57.943272 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.943273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.943283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.943361 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:57.944095 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.944650 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.945506 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.945730 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.945829 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.948617 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:57.948619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:57.948619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:57.948629 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:57.950505 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.950884 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.950885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.950885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.950941 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:57.951123 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:57.951124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:57.951124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:57.951236 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:57.951317 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.951501 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.951567 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:22:57.954128 1248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:22:57.955863 1248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:22:57.963272 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.958677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33717 (local address 127.0.0.1:54034)
0928 06:22:57.959071 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:57.959080 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:57.959202 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:57.959478 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:57.959479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:57.959480 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:57.959521 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.959523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.960842 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.960843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.961909 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.961913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.962917 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.962918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.962992 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.963029 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:22:57.963114 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:22:57.963175 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":66,"thread_start_us":126,"threads_started":1}
I20260928 06:22:57.963290 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.958960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.31.214.66:33717)
0928 06:22:57.959132 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.959138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.959152 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:22:57.959319 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.959323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.959338 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.959412 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:57.959602 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.960703 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.962059 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.962822 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.962907 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.962983 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.963144 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.963185 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260928 06:22:57.963581 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.958667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:34243 (local address 127.0.0.1:36280)
0928 06:22:57.958854 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:57.958896 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:57.958963 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:57.959407 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:57.959409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:57.959411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:57.959488 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.959490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.960804 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.960805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.962171 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.962174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.963076 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.963077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.963151 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.963154 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:22:57.963203 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:22:57.963236 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":81,"thread_start_us":52,"threads_started":1}
I20260928 06:22:57.963914 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.958768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.31.214.67:34243)
0928 06:22:57.959193 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.959197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.959206 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:22:57.959247 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.959249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.959267 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.959350 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:57.959590 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.960728 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.962264 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.962965 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.963033 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.963082 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.963318 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.963825 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":76,"threads_started":1}
I20260928 06:22:57.964236 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.958682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:41441 (local address 127.0.0.1:45488)
0928 06:22:57.959149 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:57.959159 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:57.959202 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:57.959808 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:57.959809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:57.959811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:57.959864 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.959866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.960741 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.960743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.962179 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:57.962181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.963979 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.963980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:57.964055 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.964059 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:22:57.964117 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:22:57.964169 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":51,"thread_start_us":117,"threads_started":1}
I20260928 06:22:57.964294 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:57.958940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.31.214.65:41441)
0928 06:22:57.959664 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:57.959667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:57.959675 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:22:57.959707 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:57.959708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:57.959719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:57.959798 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:57.959918 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.960615 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.963141 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:57.963887 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:57.963964 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:57.964018 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:57.964148 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:22:57.964199 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":111,"threads_started":1}
I20260928 06:22:57.964166 1525 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a (DEFAULT_TABLE table=test-table [id=702e6eec53b44491a5d7de419f7410b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:22:57.964577 1525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:57.964638 1665 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a (DEFAULT_TABLE table=test-table [id=702e6eec53b44491a5d7de419f7410b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:22:57.964952 1665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:57.964918 1385 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a (DEFAULT_TABLE table=test-table [id=702e6eec53b44491a5d7de419f7410b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:22:57.965240 1385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:57.967778 1754 tablet_bootstrap.cc:492] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: Bootstrap starting.
I20260928 06:22:57.968750 1755 tablet_bootstrap.cc:492] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: Bootstrap starting.
I20260928 06:22:57.969019 1754 tablet_bootstrap.cc:654] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:57.969080 1756 tablet_bootstrap.cc:492] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: Bootstrap starting.
I20260928 06:22:57.969411 1754 log.cc:824] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:57.969646 1755 tablet_bootstrap.cc:654] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:57.969722 1756 tablet_bootstrap.cc:654] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:57.970060 1755 log.cc:824] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:57.970118 1756 log.cc:824] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:57.970381 1754 tablet_bootstrap.cc:492] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: No bootstrap required, opened a new log
I20260928 06:22:57.970463 1754 ts_tablet_manager.cc:1397] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:22:57.970800 1755 tablet_bootstrap.cc:492] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: No bootstrap required, opened a new log
I20260928 06:22:57.970871 1755 ts_tablet_manager.cc:1397] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:22:57.972590 1754 raft_consensus.cc:348] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } }
I20260928 06:22:57.972774 1754 raft_consensus.cc:374] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:57.972810 1754 raft_consensus.cc:729] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d171167e3749b7ae35403dfa36dd0e, State: Initialized, Role: FOLLOWER
I20260928 06:22:57.972878 1755 raft_consensus.cc:348] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } }
I20260928 06:22:57.972975 1754 consensus_queue.cc:260] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } }
I20260928 06:22:57.973018 1755 raft_consensus.cc:374] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:57.973065 1755 raft_consensus.cc:729] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be5f8b6321c4e938c55591561f2510a, State: Initialized, Role: FOLLOWER
I20260928 06:22:57.973196 1755 consensus_queue.cc:260] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } }
I20260928 06:22:57.973276 1754 ts_tablet_manager.cc:1428] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:22:57.973296 1731 heartbeater.cc:503] Master 127.31.214.126:44099 was elected leader, sending a full tablet report...
I20260928 06:22:57.973450 1755 ts_tablet_manager.cc:1428] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:22:57.973534 1591 heartbeater.cc:503] Master 127.31.214.126:44099 was elected leader, sending a full tablet report...
I20260928 06:22:57.974073 1756 tablet_bootstrap.cc:492] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: No bootstrap required, opened a new log
I20260928 06:22:57.974129 1756 ts_tablet_manager.cc:1397] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 06:22:57.975718 1756 raft_consensus.cc:348] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } }
I20260928 06:22:57.975840 1756 raft_consensus.cc:374] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:57.975863 1756 raft_consensus.cc:729] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088536e2db5e4931bf5bcf0a34245284, State: Initialized, Role: FOLLOWER
I20260928 06:22:57.975941 1756 consensus_queue.cc:260] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } }
I20260928 06:22:57.976146 1756 ts_tablet_manager.cc:1428] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:22:57.976236 1451 heartbeater.cc:503] Master 127.31.214.126:44099 was elected leader, sending a full tablet report...
W20260928 06:22:58.017349 1592 tablet.cc:2369] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:22:58.095645 1762 raft_consensus.cc:482] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:22:58.095782 1762 raft_consensus.cc:504] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } }
I20260928 06:22:58.096158 1762 leader_election.cc:288] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3d171167e3749b7ae35403dfa36dd0e (127.31.214.67:34243), 3be5f8b6321c4e938c55591561f2510a (127.31.214.66:33717)
W20260928 06:22:58.096486 1452 tablet.cc:2369] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:22:58.099367 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.096536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:51767 (local address 127.31.214.67:34243)
0928 06:22:58.096588 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:58.096591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:58.096604 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:22:58.096700 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:58.096702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:58.096713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:58.096756 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:58.096981 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.097481 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.098584 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.098970 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.099058 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.099067 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:58.099245 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 06:22:58.099285 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:22:58.099355 1344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.096399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:34243 (local address 127.31.214.65:51767)
0928 06:22:58.096543 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:58.096557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:58.096609 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:58.096810 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:58.096810 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:58.096812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:58.096868 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.096870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.097565 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.097566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.098498 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.098500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.099056 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.099057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.099159 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.099163 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:22:58.099231 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:22:58.099273 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":5}
I20260928 06:22:58.099596 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.096656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:46617 (local address 127.31.214.66:33717)
0928 06:22:58.096719 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:58.096721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:58.096812 (+ 91us) server_negotiation.cc:407] Connection header received
0928 06:22:58.096898 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:58.096900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:58.096915 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:58.097013 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:22:58.097180 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.097656 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.098734 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.099126 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.099231 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.099241 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:58.099439 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:22:58.099510 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:22:58.099850 1763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.096412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33717 (local address 127.31.214.65:46617)
0928 06:22:58.096724 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:58.096734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:58.096830 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:58.097024 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:58.097025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:58.097026 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:22:58.097108 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.097109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.097745 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.097747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.098643 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.098645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.099247 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.099248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.099333 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.099337 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:22:58.099720 (+ 383us) client_negotiation.cc:240] Negotiation successful
0928 06:22:58.099764 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":163,"threads_started":1}
I20260928 06:22:58.099872 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd9ae0eaca54cfe9b3d588a8a6ac40a" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d171167e3749b7ae35403dfa36dd0e" is_pre_election: true
I20260928 06:22:58.100109 1685 raft_consensus.cc:2454] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 0.
I20260928 06:22:58.100203 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd9ae0eaca54cfe9b3d588a8a6ac40a" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be5f8b6321c4e938c55591561f2510a" is_pre_election: true
I20260928 06:22:58.100376 1545 raft_consensus.cc:2454] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 0.
I20260928 06:22:58.100430 1336 leader_election.cc:302] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088536e2db5e4931bf5bcf0a34245284, f3d171167e3749b7ae35403dfa36dd0e; no voters:
I20260928 06:22:58.100607 1762 raft_consensus.cc:2781] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:22:58.100683 1762 raft_consensus.cc:482] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:22:58.100703 1762 raft_consensus.cc:3037] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:58.101415 1762 raft_consensus.cc:504] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } }
I20260928 06:22:58.101577 1762 leader_election.cc:288] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 election: Requested vote from peers f3d171167e3749b7ae35403dfa36dd0e (127.31.214.67:34243), 3be5f8b6321c4e938c55591561f2510a (127.31.214.66:33717)
I20260928 06:22:58.101760 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd9ae0eaca54cfe9b3d588a8a6ac40a" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d171167e3749b7ae35403dfa36dd0e"
I20260928 06:22:58.101763 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd9ae0eaca54cfe9b3d588a8a6ac40a" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be5f8b6321c4e938c55591561f2510a"
I20260928 06:22:58.101841 1685 raft_consensus.cc:3037] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:58.101841 1545 raft_consensus.cc:3037] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:58.102780 1545 raft_consensus.cc:2454] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 1.
I20260928 06:22:58.102780 1685 raft_consensus.cc:2454] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 1.
I20260928 06:22:58.103401 1337 leader_election.cc:302] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088536e2db5e4931bf5bcf0a34245284, 3be5f8b6321c4e938c55591561f2510a; no voters:
I20260928 06:22:58.103545 1762 raft_consensus.cc:2781] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:22:58.103734 1762 raft_consensus.cc:686] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 1 LEADER]: Becoming Leader. State: Replica: 088536e2db5e4931bf5bcf0a34245284, State: Running, Role: LEADER
I20260928 06:22:58.103812 1762 consensus_queue.cc:237] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } }
I20260928 06:22:58.104748 1248 catalog_manager.cc:5495] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088536e2db5e4931bf5bcf0a34245284 (127.31.214.65). New cstate: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } health_report { overall_health: HEALTHY } } }
I20260928 06:22:58.117311 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.114580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.31.214.65:41441)
0928 06:22:58.114657 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:58.114659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:58.114670 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:22:58.114794 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:58.114795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:58.114826 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:58.114903 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:22:58.115068 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.115568 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.116403 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.116606 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.116685 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.116911 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:22:58.117023 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:22:58.117136 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:58.117192 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:22:58.117214 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:22:58.119359 1545 raft_consensus.cc:1261] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Refusing update from remote peer 088536e2db5e4931bf5bcf0a34245284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:22:58.119365 1685 raft_consensus.cc:1261] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Refusing update from remote peer 088536e2db5e4931bf5bcf0a34245284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:22:58.119856 1764 consensus_queue.cc:1035] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:22:58.120016 1762 consensus_queue.cc:1035] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:22:58.122601 1768 mvcc.cc:205] Tried to move back new op lower bound from 7334201663974121472 to 7334201663913938944. Current Snapshot: MvccSnapshot[applied={T|T < 7334201663974121472}]
I20260928 06:22:58.126336 1247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:22:58.126430 1247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:22:58.127269 1247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf: Sending DeleteTablet for 3 replicas of tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a
I20260928 06:22:58.127672 1665 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:22:58 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:36280
I20260928 06:22:58.127765 1525 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:22:58 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:54034
I20260928 06:22:58.127784 1385 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:22:58 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:45488
I20260928 06:22:58.128098 1780 tablet_replica.cc:330] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: stopping tablet replica
I20260928 06:22:58.128221 1780 raft_consensus.cc:2229] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:22:58.128175 1247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:22:58.128315 1780 raft_consensus.cc:2258] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 06:22:58.128324 1247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:22:58.128700 1781 tablet_replica.cc:330] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: stopping tablet replica
I20260928 06:22:58.128841 1781 raft_consensus.cc:2229] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:22:58.128998 1781 raft_consensus.cc:2258] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:22:58.129431 1780 ts_tablet_manager.cc:1905] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:22:58.129853 1781 ts_tablet_manager.cc:1905] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:22:58.130018 1385 tablet_service.cc:1467] Processing CreateTablet for tablet d9197eb439c54b028c7631771c6a1317 (DEFAULT_TABLE table=test-table [id=486e396011e1476aaf9c1376e1808b5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:22:58.130205 1385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9197eb439c54b028c7631771c6a1317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:58.130288 1525 tablet_service.cc:1467] Processing CreateTablet for tablet d9197eb439c54b028c7631771c6a1317 (DEFAULT_TABLE table=test-table [id=486e396011e1476aaf9c1376e1808b5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:22:58.130386 1525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9197eb439c54b028c7631771c6a1317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:58.131652 1756 tablet_bootstrap.cc:492] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: Bootstrap starting.
I20260928 06:22:58.131953 1665 tablet_service.cc:1467] Processing CreateTablet for tablet d9197eb439c54b028c7631771c6a1317 (DEFAULT_TABLE table=test-table [id=486e396011e1476aaf9c1376e1808b5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:22:58.132097 1665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9197eb439c54b028c7631771c6a1317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:58.132107 1756 tablet_bootstrap.cc:654] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:58.132193 1780 ts_tablet_manager.cc:1918] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:22:58.132731 1756 tablet_bootstrap.cc:492] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: No bootstrap required, opened a new log
I20260928 06:22:58.132786 1756 ts_tablet_manager.cc:1397] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:58.132441 1755 tablet_bootstrap.cc:492] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: Bootstrap starting.
I20260928 06:22:58.132942 1756 raft_consensus.cc:348] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.133011 1756 raft_consensus.cc:374] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:58.133033 1756 raft_consensus.cc:729] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088536e2db5e4931bf5bcf0a34245284, State: Initialized, Role: FOLLOWER
I20260928 06:22:58.133077 1756 consensus_queue.cc:260] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.133157 1756 ts_tablet_manager.cc:1428] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:22:58.133350 1755 tablet_bootstrap.cc:654] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:58.133920 1780 log.cc:1196] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/8bd9ae0eaca54cfe9b3d588a8a6ac40a
I20260928 06:22:58.134058 1754 tablet_bootstrap.cc:492] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: Bootstrap starting.
I20260928 06:22:58.134392 1780 ts_tablet_manager.cc:1939] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 3be5f8b6321c4e938c55591561f2510a: Deleting consensus metadata
I20260928 06:22:58.134495 1754 tablet_bootstrap.cc:654] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:58.134892 1755 tablet_bootstrap.cc:492] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: No bootstrap required, opened a new log
I20260928 06:22:58.134943 1755 ts_tablet_manager.cc:1397] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 06:22:58.135033 1233 catalog_manager.cc:4841] TS 3be5f8b6321c4e938c55591561f2510a (127.31.214.66:33717): tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a (table test-table [id=702e6eec53b44491a5d7de419f7410b0]) successfully deleted
I20260928 06:22:58.135107 1755 raft_consensus.cc:348] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.135214 1755 raft_consensus.cc:374] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:58.135257 1755 raft_consensus.cc:729] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be5f8b6321c4e938c55591561f2510a, State: Initialized, Role: FOLLOWER
I20260928 06:22:58.135308 1755 consensus_queue.cc:260] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.135417 1755 ts_tablet_manager.cc:1428] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:58.136586 1781 ts_tablet_manager.cc:1918] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:22:58.136686 1781 log.cc:1196] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/8bd9ae0eaca54cfe9b3d588a8a6ac40a
I20260928 06:22:58.136919 1781 ts_tablet_manager.cc:1939] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P 088536e2db5e4931bf5bcf0a34245284: Deleting consensus metadata
I20260928 06:22:58.137465 1235 catalog_manager.cc:4841] TS 088536e2db5e4931bf5bcf0a34245284 (127.31.214.65:41441): tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a (table test-table [id=702e6eec53b44491a5d7de419f7410b0]) successfully deleted
I20260928 06:22:58.138106 1779 tablet_replica.cc:330] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: stopping tablet replica
I20260928 06:22:58.138247 1779 raft_consensus.cc:2229] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:22:58.138386 1779 raft_consensus.cc:2258] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:22:58.138441 1754 tablet_bootstrap.cc:492] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: No bootstrap required, opened a new log
I20260928 06:22:58.138497 1754 ts_tablet_manager.cc:1397] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:22:58.138657 1754 raft_consensus.cc:348] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.138746 1754 raft_consensus.cc:374] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:58.138779 1754 raft_consensus.cc:729] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d171167e3749b7ae35403dfa36dd0e, State: Initialized, Role: FOLLOWER
I20260928 06:22:58.138840 1754 consensus_queue.cc:260] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.138931 1754 ts_tablet_manager.cc:1428] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:22:58.139034 1779 ts_tablet_manager.cc:1905] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:22:58.140645 1779 ts_tablet_manager.cc:1918] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:22:58.140759 1779 log.cc:1196] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/8bd9ae0eaca54cfe9b3d588a8a6ac40a
I20260928 06:22:58.141050 1779 ts_tablet_manager.cc:1939] T 8bd9ae0eaca54cfe9b3d588a8a6ac40a P f3d171167e3749b7ae35403dfa36dd0e: Deleting consensus metadata
I20260928 06:22:58.141537 1232 catalog_manager.cc:4841] TS f3d171167e3749b7ae35403dfa36dd0e (127.31.214.67:34243): tablet 8bd9ae0eaca54cfe9b3d588a8a6ac40a (table test-table [id=702e6eec53b44491a5d7de419f7410b0]) successfully deleted
W20260928 06:22:58.171198 1732 tablet.cc:2369] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:22:58.200803 1764 raft_consensus.cc:482] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:22:58.200912 1764 raft_consensus.cc:504] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.201115 1764 leader_election.cc:288] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3be5f8b6321c4e938c55591561f2510a (127.31.214.66:33717), f3d171167e3749b7ae35403dfa36dd0e (127.31.214.67:34243)
I20260928 06:22:58.201437 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9197eb439c54b028c7631771c6a1317" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be5f8b6321c4e938c55591561f2510a" is_pre_election: true
I20260928 06:22:58.201426 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9197eb439c54b028c7631771c6a1317" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d171167e3749b7ae35403dfa36dd0e" is_pre_election: true
I20260928 06:22:58.201558 1685 raft_consensus.cc:2454] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 0.
I20260928 06:22:58.201560 1545 raft_consensus.cc:2454] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 0.
I20260928 06:22:58.201754 1337 leader_election.cc:302] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088536e2db5e4931bf5bcf0a34245284, 3be5f8b6321c4e938c55591561f2510a; no voters:
I20260928 06:22:58.201881 1764 raft_consensus.cc:2781] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:22:58.201948 1764 raft_consensus.cc:482] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:22:58.202004 1764 raft_consensus.cc:3037] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:58.202831 1764 raft_consensus.cc:504] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.203012 1764 leader_election.cc:288] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 election: Requested vote from peers 3be5f8b6321c4e938c55591561f2510a (127.31.214.66:33717), f3d171167e3749b7ae35403dfa36dd0e (127.31.214.67:34243)
I20260928 06:22:58.203181 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9197eb439c54b028c7631771c6a1317" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be5f8b6321c4e938c55591561f2510a"
I20260928 06:22:58.203205 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9197eb439c54b028c7631771c6a1317" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d171167e3749b7ae35403dfa36dd0e"
I20260928 06:22:58.203270 1545 raft_consensus.cc:3037] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:58.203279 1685 raft_consensus.cc:3037] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:58.204104 1545 raft_consensus.cc:2454] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 1.
I20260928 06:22:58.204172 1685 raft_consensus.cc:2454] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 1.
I20260928 06:22:58.204355 1336 leader_election.cc:302] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088536e2db5e4931bf5bcf0a34245284, f3d171167e3749b7ae35403dfa36dd0e; no voters:
I20260928 06:22:58.204519 1764 raft_consensus.cc:2781] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:22:58.204597 1764 raft_consensus.cc:686] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 1 LEADER]: Becoming Leader. State: Replica: 088536e2db5e4931bf5bcf0a34245284, State: Running, Role: LEADER
I20260928 06:22:58.204658 1764 consensus_queue.cc:237] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.205420 1247 catalog_manager.cc:5495] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088536e2db5e4931bf5bcf0a34245284 (127.31.214.65). New cstate: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:22:58 dist-test-slave-q03f krb5kdc[1203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:58.258786 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.250769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44099 (local address 127.0.0.1:49692)
0928 06:22:58.250932 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:58.250952 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:58.251009 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:58.251264 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:58.251266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:58.251659 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:58.251717 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.251720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.252443 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.252443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.253070 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.253073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.253491 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.253492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.253543 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.254351 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:58.254357 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:58.255896 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:58.257562 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:58.257571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:58.257574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:58.257798 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:58.258067 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:58.258068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:58.258068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:58.258133 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:58.258379 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:58.258387 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:58.258435 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:22:58.258494 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:22:58.258659 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260928 06:22:58.258790 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.251003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.31.214.126:44099)
0928 06:22:58.251063 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:58.251065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:58.251083 (+ 18us) server_negotiation.cc:407] Connection header received
0928 06:22:58.251112 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:58.251112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:58.251132 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:58.251206 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:58.251837 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.252356 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.253189 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.253403 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.253473 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.255996 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:58.255997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:58.255998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:58.256005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:58.257466 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:58.257873 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:58.257874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:58.257875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:58.257931 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:58.258196 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:58.258197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:58.258197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:58.258316 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:58.258370 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:58.258528 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:22:58.258711 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":31}
I20260928 06:22:58.261430 1247 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:49692:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:22:58.261710 1247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:22:58.263481 1665 tablet_service.cc:1467] Processing CreateTablet for tablet fc1bb53eef3d467cb8fa02417874dae9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acec6f8454024c20932c6180fc08bd53]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:22:58.263600 1385 tablet_service.cc:1467] Processing CreateTablet for tablet fc1bb53eef3d467cb8fa02417874dae9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acec6f8454024c20932c6180fc08bd53]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:22:58.263624 1525 tablet_service.cc:1467] Processing CreateTablet for tablet fc1bb53eef3d467cb8fa02417874dae9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acec6f8454024c20932c6180fc08bd53]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:22:58.263691 1665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1bb53eef3d467cb8fa02417874dae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:58.263784 1525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1bb53eef3d467cb8fa02417874dae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:58.263809 1385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1bb53eef3d467cb8fa02417874dae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:58.265388 1756 tablet_bootstrap.cc:492] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: Bootstrap starting.
I20260928 06:22:58.265691 1754 tablet_bootstrap.cc:492] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: Bootstrap starting.
I20260928 06:22:58.265687 1755 tablet_bootstrap.cc:492] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: Bootstrap starting.
I20260928 06:22:58.266006 1756 tablet_bootstrap.cc:654] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:58.266168 1754 tablet_bootstrap.cc:654] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:58.266250 1755 tablet_bootstrap.cc:654] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:58.266721 1756 tablet_bootstrap.cc:492] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: No bootstrap required, opened a new log
I20260928 06:22:58.266780 1756 ts_tablet_manager.cc:1397] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:58.266904 1756 raft_consensus.cc:348] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.266944 1755 tablet_bootstrap.cc:492] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: No bootstrap required, opened a new log
I20260928 06:22:58.267004 1755 ts_tablet_manager.cc:1397] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:22:58.267032 1756 raft_consensus.cc:374] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:58.267066 1756 raft_consensus.cc:729] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088536e2db5e4931bf5bcf0a34245284, State: Initialized, Role: FOLLOWER
I20260928 06:22:58.267132 1756 consensus_queue.cc:260] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.267195 1755 raft_consensus.cc:348] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.267241 1756 ts_tablet_manager.cc:1428] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:58.267270 1755 raft_consensus.cc:374] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:58.267295 1755 raft_consensus.cc:729] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be5f8b6321c4e938c55591561f2510a, State: Initialized, Role: FOLLOWER
I20260928 06:22:58.267267 1764 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.267347 1755 consensus_queue.cc:260] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.267376 1764 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:22:58.267448 1755 ts_tablet_manager.cc:1428] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:22:58.267611 1754 tablet_bootstrap.cc:492] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: No bootstrap required, opened a new log
I20260928 06:22:58.267611 1761 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.267665 1754 ts_tablet_manager.cc:1397] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:22:58.267690 1761 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:22:58.267828 1754 raft_consensus.cc:348] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.267903 1754 raft_consensus.cc:374] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:58.267923 1754 raft_consensus.cc:729] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d171167e3749b7ae35403dfa36dd0e, State: Initialized, Role: FOLLOWER
I20260928 06:22:58.267964 1754 consensus_queue.cc:260] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.268074 1754 ts_tablet_manager.cc:1428] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:58.268137 1760 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.268262 1760 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:22:58.279572 1764 raft_consensus.cc:482] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:22:58.279670 1764 raft_consensus.cc:504] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.279866 1764 leader_election.cc:288] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3be5f8b6321c4e938c55591561f2510a (127.31.214.66:33717), f3d171167e3749b7ae35403dfa36dd0e (127.31.214.67:34243)
I20260928 06:22:58.280092 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1bb53eef3d467cb8fa02417874dae9" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be5f8b6321c4e938c55591561f2510a" is_pre_election: true
I20260928 06:22:58.280124 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1bb53eef3d467cb8fa02417874dae9" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d171167e3749b7ae35403dfa36dd0e" is_pre_election: true
I20260928 06:22:58.280210 1685 raft_consensus.cc:2454] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 0.
I20260928 06:22:58.280211 1545 raft_consensus.cc:2454] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 0.
I20260928 06:22:58.280375 1336 leader_election.cc:302] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088536e2db5e4931bf5bcf0a34245284, f3d171167e3749b7ae35403dfa36dd0e; no voters:
I20260928 06:22:58.280519 1764 raft_consensus.cc:2781] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:22:58.280558 1764 raft_consensus.cc:482] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:22:58.280596 1764 raft_consensus.cc:3037] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:58.281368 1764 raft_consensus.cc:504] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.281545 1764 leader_election.cc:288] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 election: Requested vote from peers 3be5f8b6321c4e938c55591561f2510a (127.31.214.66:33717), f3d171167e3749b7ae35403dfa36dd0e (127.31.214.67:34243)
I20260928 06:22:58.281690 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1bb53eef3d467cb8fa02417874dae9" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be5f8b6321c4e938c55591561f2510a"
I20260928 06:22:58.281782 1545 raft_consensus.cc:3037] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:58.281806 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1bb53eef3d467cb8fa02417874dae9" candidate_uuid: "088536e2db5e4931bf5bcf0a34245284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d171167e3749b7ae35403dfa36dd0e"
I20260928 06:22:58.281896 1685 raft_consensus.cc:3037] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:58.282685 1545 raft_consensus.cc:2454] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 1.
I20260928 06:22:58.282773 1685 raft_consensus.cc:2454] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088536e2db5e4931bf5bcf0a34245284 in term 1.
I20260928 06:22:58.282910 1337 leader_election.cc:302] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088536e2db5e4931bf5bcf0a34245284, 3be5f8b6321c4e938c55591561f2510a; no voters:
I20260928 06:22:58.283020 1764 raft_consensus.cc:2781] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:22:58.283083 1764 raft_consensus.cc:686] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [term 1 LEADER]: Becoming Leader. State: Replica: 088536e2db5e4931bf5bcf0a34245284, State: Running, Role: LEADER
I20260928 06:22:58.283149 1764 consensus_queue.cc:237] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } }
I20260928 06:22:58.283524 1776 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: TxnStatusTablet state changed. Reason: New leader 088536e2db5e4931bf5bcf0a34245284. Latest consensus state: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.283632 1776 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:22:58.283839 1796 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:22:58.283984 1796 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:22:58.284013 1246 catalog_manager.cc:5495] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088536e2db5e4931bf5bcf0a34245284 (127.31.214.65). New cstate: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } health_report { overall_health: UNKNOWN } } }
I20260928 06:22:58.293231 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.290454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:41441 (local address 127.0.0.1:55232)
0928 06:22:58.290539 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:58.290551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:58.290609 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:58.290885 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:58.290886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:58.290887 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:22:58.290960 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.290962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.291690 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.291691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.292351 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.292354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.292726 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.292727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.292818 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.292834 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:22:58.293055 (+ 221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:22:58.293065 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 06:22:58.293108 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:22:58.293161 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:22:58.293217 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.290691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.31.214.65:41441)
0928 06:22:58.290754 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:58.290756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:58.290763 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:22:58.290794 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:58.290795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:58.290805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:58.290871 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:22:58.291055 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.291587 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.292448 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.292652 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.292727 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.292934 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:22:58.292976 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:22:58.293033 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:58.293111 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:22:58.293137 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:22:58.294240 1545 raft_consensus.cc:1261] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Refusing update from remote peer 088536e2db5e4931bf5bcf0a34245284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:22:58.294240 1685 raft_consensus.cc:1261] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Refusing update from remote peer 088536e2db5e4931bf5bcf0a34245284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:22:58.294467 1764 consensus_queue.cc:1035] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:22:58.294620 1764 consensus_queue.cc:1035] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:22:58.295116 1760 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: TxnStatusTablet state changed. Reason: New leader 088536e2db5e4931bf5bcf0a34245284. Latest consensus state: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.295182 1761 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: TxnStatusTablet state changed. Reason: New leader 088536e2db5e4931bf5bcf0a34245284. Latest consensus state: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.295253 1760 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:22:58.295264 1761 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:22:58.295960 1777 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.296063 1777 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:22:58.296090 1760 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.296094 1761 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.296126 1777 tablet_replica.cc:439] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "088536e2db5e4931bf5bcf0a34245284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088536e2db5e4931bf5bcf0a34245284" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41441 } } peers { permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 } } peers { permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 } } }
I20260928 06:22:58.296170 1760 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P f3d171167e3749b7ae35403dfa36dd0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:22:58.296170 1761 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P 3be5f8b6321c4e938c55591561f2510a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:22:58.296175 1777 tablet_replica.cc:442] T fc1bb53eef3d467cb8fa02417874dae9 P 088536e2db5e4931bf5bcf0a34245284: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:22:58.296249 1767 mvcc.cc:205] Tried to move back new op lower bound from 7334201664691290112 to 7334201664648204288. Current Snapshot: MvccSnapshot[applied={T|T < 7334201664691290112}]
I20260928 06:22:58.302682 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.300267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:38011 (local address 127.31.214.65:41441)
0928 06:22:58.300304 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:58.300306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:58.300444 (+ 138us) server_negotiation.cc:407] Connection header received
0928 06:22:58.300507 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:58.300508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:58.300517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:58.300578 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:22:58.300679 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.301156 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.301908 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.302146 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.302232 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.302400 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:22:58.302438 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:22:58.302498 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:58.302557 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:22:58.302597 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":16}
I20260928 06:22:58.302688 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.300208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:41441 (local address 127.31.214.65:38011)
0928 06:22:58.300370 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:58.300388 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:58.300461 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:58.300581 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:58.300581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:58.300583 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:22:58.300632 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.300634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.301241 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.301242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.301825 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:58.301826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.302233 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.302234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:58.302284 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.302300 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:22:58.302498 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:22:58.302500 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:22:58.302552 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:22:58.302616 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":4,"thread_start_us":104,"threads_started":1}
I20260928 06:22:58.304832 1685 raft_consensus.cc:1261] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Refusing update from remote peer 088536e2db5e4931bf5bcf0a34245284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:22:58.304888 1545 raft_consensus.cc:1261] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Refusing update from remote peer 088536e2db5e4931bf5bcf0a34245284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:22:58.305032 1762 consensus_queue.cc:1035] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3d171167e3749b7ae35403dfa36dd0e" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:22:58.305160 1764 consensus_queue.cc:1035] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be5f8b6321c4e938c55591561f2510a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:22:58.319592 1247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:22:58.319701 1247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:22:58.320341 1247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74c0c2cd271e419dbc85856a61fe1fcf: Sending DeleteTablet for 3 replicas of tablet d9197eb439c54b028c7631771c6a1317
I20260928 06:22:58.320624 1665 tablet_service.cc:1514] Processing DeleteTablet for tablet d9197eb439c54b028c7631771c6a1317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:22:58 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:36280
I20260928 06:22:58.320737 1384 tablet_service.cc:1514] Processing DeleteTablet for tablet d9197eb439c54b028c7631771c6a1317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:22:58 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:45488
I20260928 06:22:58.320732 1525 tablet_service.cc:1514] Processing DeleteTablet for tablet d9197eb439c54b028c7631771c6a1317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:22:58 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:54034
I20260928 06:22:58.320824 1781 tablet_replica.cc:330] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: stopping tablet replica
I20260928 06:22:58.320851 1779 tablet_replica.cc:330] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: stopping tablet replica
I20260928 06:22:58.320879 1781 raft_consensus.cc:2229] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:22:58.320940 1781 raft_consensus.cc:2258] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:22:58.320928 1779 raft_consensus.cc:2229] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:22:58.321008 1779 raft_consensus.cc:2258] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:22:58.321085 1780 tablet_replica.cc:330] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: stopping tablet replica
I20260928 06:22:58.321142 1780 raft_consensus.cc:2229] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:22:58.321187 1780 raft_consensus.cc:2258] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:22:58.321563 1781 ts_tablet_manager.cc:1905] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:22:58.321558 1779 ts_tablet_manager.cc:1905] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:22:58.323431 1781 ts_tablet_manager.cc:1918] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:22:58.323458 1780 ts_tablet_manager.cc:1905] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:22:58.323500 1781 log.cc:1196] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/d9197eb439c54b028c7631771c6a1317
I20260928 06:22:58.323750 1781 ts_tablet_manager.cc:1939] T d9197eb439c54b028c7631771c6a1317 P 088536e2db5e4931bf5bcf0a34245284: Deleting consensus metadata
I20260928 06:22:58.324303 1235 catalog_manager.cc:4841] TS 088536e2db5e4931bf5bcf0a34245284 (127.31.214.65:41441): tablet d9197eb439c54b028c7631771c6a1317 (table test-table [id=486e396011e1476aaf9c1376e1808b5d]) successfully deleted
I20260928 06:22:58.325178 1780 ts_tablet_manager.cc:1918] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:22:58.325218 1780 log.cc:1196] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/d9197eb439c54b028c7631771c6a1317
I20260928 06:22:58.325412 1780 ts_tablet_manager.cc:1939] T d9197eb439c54b028c7631771c6a1317 P 3be5f8b6321c4e938c55591561f2510a: Deleting consensus metadata
I20260928 06:22:58.325769 1779 ts_tablet_manager.cc:1918] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:22:58.325825 1779 log.cc:1196] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/d9197eb439c54b028c7631771c6a1317
I20260928 06:22:58.326058 1779 ts_tablet_manager.cc:1939] T d9197eb439c54b028c7631771c6a1317 P f3d171167e3749b7ae35403dfa36dd0e: Deleting consensus metadata
I20260928 06:22:58.326347 1232 catalog_manager.cc:4841] TS f3d171167e3749b7ae35403dfa36dd0e (127.31.214.67:34243): tablet d9197eb439c54b028c7631771c6a1317 (table test-table [id=486e396011e1476aaf9c1376e1808b5d]) successfully deleted
I20260928 06:22:58.327095 1233 catalog_manager.cc:4841] TS 3be5f8b6321c4e938c55591561f2510a (127.31.214.66:33717): tablet d9197eb439c54b028c7631771c6a1317 (table test-table [id=486e396011e1476aaf9c1376e1808b5d]) successfully deleted
Sep 28 06:22:58 dist-test-slave-q03f krb5kdc[1203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:22:58.333753 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.322258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.31.214.65:41441)
0928 06:22:58.323193 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:58.323196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:58.323513 (+ 317us) server_negotiation.cc:407] Connection header received
0928 06:22:58.323629 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:58.323630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:58.323641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:58.323673 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:58.324468 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.325289 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.326493 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.326826 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.326907 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.330284 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:58.330296 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:58.330297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:58.330306 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:58.332443 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:58.332846 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:58.332850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:58.332851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:58.332926 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:58.333129 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:58.333130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:58.333131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:58.333333 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:58.333495 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:58.333537 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 06:22:58.333629 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907}
W20260928 06:22:58.334016 1445 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55238
I20260928 06:22:58.343293 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:58.335134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.31.214.126:44099)
0928 06:22:58.335205 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:58.335207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:58.336235 (+ 1028us) server_negotiation.cc:407] Connection header received
0928 06:22:58.336285 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:58.336286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:58.336300 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:58.336373 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:58.336847 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.337746 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.338599 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:58.338947 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:58.339041 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:58.340272 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:58.340273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:58.340273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:58.340281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:58.341904 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:58.342460 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:58.342461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:58.342461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:58.342513 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:58.342791 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:58.342792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:58.342792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:58.342902 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:58.342977 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:58.343161 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 06:22:58.343222 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 06:22:58.343588 1247 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40582
I20260928 06:22:58.344254 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 1320
I20260928 06:22:58.351631 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 1460
I20260928 06:22:58.357362 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 1600
I20260928 06:22:58.362025 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 1216
2026-09-28T06:22:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2373 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:22:58 dist-test-slave-q03f krb5kdc[1812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:22:58 dist-test-slave-q03f krb5kdc[1812](info): set up 2 sockets
Sep 28 06:22:58 dist-test-slave-q03f krb5kdc[1812](info): commencing operation
krb5kdc: starting...
W20260928 06:22:59.425845 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:22:59 dist-test-slave-q03f krb5kdc[1812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:22:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:22:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:22:59.557381 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:33903
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:33903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:59.641801 1825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:59.642076 1825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:59.642103 1825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:59.643316 1825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:22:59.643378 1825 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:22:59.643391 1825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:59.643405 1825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:59.643417 1825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:22:59.643430 1825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:22:59.645018 1825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:33903
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:33903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:59.645258 1825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:59.645471 1825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:59.650856 1831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:59.650871 1830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:59.650931 1833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:59.651566 1825 server_base.cc:1034] running on GCE node
I20260928 06:22:59.651880 1825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:59.652326 1825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:59.653571 1825 hybrid_clock.cc:648] HybridClock initialized: now 1790576579653550 us; error 38 us; skew 500 ppm
Sep 28 06:22:59 dist-test-slave-q03f krb5kdc[1812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576579, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:59.656461 1825 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:22:59.657065 1825 webserver.cc:460] Webserver started at http://127.31.214.126:36251/ using document root <none> and password file <none>
I20260928 06:22:59.657347 1825 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:59.657424 1825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:59.657594 1825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:59.658850 1825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "414053a253b4476d8f4ce95ec44e8cb9"
format_stamp: "Formatted at 2026-09-28 06:22:59 on dist-test-slave-q03f"
server_key: "d523b60b5e9c8e124962f9a92e98dcac"
server_key_iv: "b744d92dd46fd3ac29b776ecf754a85a"
server_key_version: "encryptionkey@0"
I20260928 06:22:59.659317 1825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "414053a253b4476d8f4ce95ec44e8cb9"
format_stamp: "Formatted at 2026-09-28 06:22:59 on dist-test-slave-q03f"
server_key: "d523b60b5e9c8e124962f9a92e98dcac"
server_key_iv: "b744d92dd46fd3ac29b776ecf754a85a"
server_key_version: "encryptionkey@0"
I20260928 06:22:59.660925 1825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:22:59.662091 1840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:59.662344 1825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:59.662473 1825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "414053a253b4476d8f4ce95ec44e8cb9"
format_stamp: "Formatted at 2026-09-28 06:22:59 on dist-test-slave-q03f"
server_key: "d523b60b5e9c8e124962f9a92e98dcac"
server_key_iv: "b744d92dd46fd3ac29b776ecf754a85a"
server_key_version: "encryptionkey@0"
I20260928 06:22:59.662566 1825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:59.690941 1825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:59.693789 1825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:59.694025 1825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:59.699657 1825 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:33903
I20260928 06:22:59.699761 1902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:33903 every 8 connection(s)
I20260928 06:22:59.700152 1825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:22:59.700941 1903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:22:59.702992 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 1825
I20260928 06:22:59.703073 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:22:59.703270 32601 external_mini_cluster.cc:1428] Setting key ff099c2174b6a4386348d38304b2f686
I20260928 06:22:59.703788 1903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9: Bootstrap starting.
I20260928 06:22:59.704478 1903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9: Neither blocks nor log segments found. Creating new log.
I20260928 06:22:59.704814 1903 log.cc:824] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:22:59.705663 1903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9: No bootstrap required, opened a new log
I20260928 06:22:59.707772 1903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414053a253b4476d8f4ce95ec44e8cb9" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 33903 } }
I20260928 06:22:59.707947 1903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:22:59.708004 1903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414053a253b4476d8f4ce95ec44e8cb9, State: Initialized, Role: FOLLOWER
I20260928 06:22:59.708144 1903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414053a253b4476d8f4ce95ec44e8cb9" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 33903 } }
I20260928 06:22:59.708236 1903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:22:59.708281 1903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:22:59.708334 1903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:22:59.708878 1903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414053a253b4476d8f4ce95ec44e8cb9" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 33903 } }
I20260928 06:22:59.708999 1903 leader_election.cc:302] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 414053a253b4476d8f4ce95ec44e8cb9; no voters:
I20260928 06:22:59.709215 1903 leader_election.cc:288] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:22:59.709244 1908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:22:59.709367 1908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [term 1 LEADER]: Becoming Leader. State: Replica: 414053a253b4476d8f4ce95ec44e8cb9, State: Running, Role: LEADER
I20260928 06:22:59.709451 1908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414053a253b4476d8f4ce95ec44e8cb9" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 33903 } }
I20260928 06:22:59.709515 1903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:22:59.709771 1908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 414053a253b4476d8f4ce95ec44e8cb9. Latest consensus state: current_term: 1 leader_uuid: "414053a253b4476d8f4ce95ec44e8cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414053a253b4476d8f4ce95ec44e8cb9" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 33903 } } }
I20260928 06:22:59.709851 1908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:22:59.709784 1910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "414053a253b4476d8f4ce95ec44e8cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414053a253b4476d8f4ce95ec44e8cb9" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 33903 } } }
I20260928 06:22:59.710069 1910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9 [sys.catalog]: This master's current role is: LEADER
Sep 28 06:22:59 dist-test-slave-q03f krb5kdc[1812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:59.710101 1912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:22:59.710660 1912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:22:59.714427 1912 catalog_manager.cc:1347] Generated new cluster ID: 41baa02a7f31415db8d2fc172db551e6
I20260928 06:22:59.714428 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:59.703837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.31.214.126:33903)
0928 06:22:59.704028 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:59.704051 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:59.704089 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:22:59.704579 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:59.704589 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:59.704656 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:59.704815 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:59.705615 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.706649 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.707413 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.707841 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.707963 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:59.710637 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:59.710641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:59.710641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:59.710654 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:59.712749 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:59.713134 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:59.713137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:59.713137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:59.713236 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:59.713450 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:59.713451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:59.713451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:59.713697 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:59.713885 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:59.714032 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 06:22:59.714174 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260928 06:22:59.714505 1912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:22:59.719727 1912 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:22:59.720461 1912 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:22:59.722946 1912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9: Generated new TSK 0
I20260928 06:22:59.723121 1912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:22:59.758283 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:33903
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:22:59.853662 1929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:22:59.853873 1929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:22:59.853902 1929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:22:59.855120 1929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:22:59.855167 1929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:22:59.855188 1929 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:22:59.855201 1929 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:22:59.855217 1929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:22:59.856761 1929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.214.126:33903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:22:59.857021 1929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:22:59.857268 1929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:22:59.862435 1935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:22:59.862547 1937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:59.862771 1929 server_base.cc:1034] running on GCE node
W20260928 06:22:59.862826 1934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:22:59.863070 1929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:22:59.863320 1929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:22:59.864568 1929 hybrid_clock.cc:648] HybridClock initialized: now 1790576579864542 us; error 37 us; skew 500 ppm
Sep 28 06:22:59 dist-test-slave-q03f krb5kdc[1812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576579, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:22:59.867610 1929 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:22:59.868155 1929 webserver.cc:460] Webserver started at http://127.31.214.65:46077/ using document root <none> and password file <none>
I20260928 06:22:59.868420 1929 fs_manager.cc:359] Metadata directory not provided
I20260928 06:22:59.868500 1929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:22:59.868656 1929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:22:59.869827 1929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "b5349192348940f99ca4e6f57033b584"
format_stamp: "Formatted at 2026-09-28 06:22:59 on dist-test-slave-q03f"
server_key: "46695166471e39d21304fa93f2ec0cd7"
server_key_iv: "9c6b5a4c54681092959a98339ba1b6c2"
server_key_version: "encryptionkey@0"
I20260928 06:22:59.870294 1929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "b5349192348940f99ca4e6f57033b584"
format_stamp: "Formatted at 2026-09-28 06:22:59 on dist-test-slave-q03f"
server_key: "46695166471e39d21304fa93f2ec0cd7"
server_key_iv: "9c6b5a4c54681092959a98339ba1b6c2"
server_key_version: "encryptionkey@0"
I20260928 06:22:59.871811 1929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:59.872701 1944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:59.872985 1929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:22:59.873121 1929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "b5349192348940f99ca4e6f57033b584"
format_stamp: "Formatted at 2026-09-28 06:22:59 on dist-test-slave-q03f"
server_key: "46695166471e39d21304fa93f2ec0cd7"
server_key_iv: "9c6b5a4c54681092959a98339ba1b6c2"
server_key_version: "encryptionkey@0"
I20260928 06:22:59.873208 1929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:22:59.895291 1929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:22:59.897749 1929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:22:59.897989 1929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:22:59.898666 1929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:22:59.898732 1929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:22:59.898778 1929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:22:59.898802 1929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:22:59 dist-test-slave-q03f krb5kdc[1812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576579, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:22:59.907006 1929 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:43391
I20260928 06:22:59.907114 2059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:43391 every 8 connection(s)
I20260928 06:22:59.907486 1929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:22:59.909621 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:59.899220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:58995 (local address 127.31.214.126:33903)
0928 06:22:59.899293 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:59.899296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:59.899522 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:22:59.900032 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:59.900033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:59.900048 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:59.900149 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:59.901088 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.902097 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.903278 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.903509 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.903610 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:59.906590 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:59.906592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:59.906592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:59.906599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:59.908292 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:59.908744 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:59.908745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:59.908746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:59.908800 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:59.909023 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:59.909024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:59.909024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:59.909140 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:59.909190 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:59.909399 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 06:22:59.909507 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:22:59.909829 1953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:59.899191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:33903 (local address 127.31.214.65:58995)
0928 06:22:59.899377 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:59.899419 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:59.899925 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:59.900212 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:59.900216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:59.900608 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:59.900946 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:59.900955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.902206 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.902207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:59.903179 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:59.903181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.903636 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.903637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:59.903765 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:59.904352 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:59.904365 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:59.906422 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:59.908386 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:59.908395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:59.908398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:59.908656 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:59.908863 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:59.908863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:59.908863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:59.908967 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:59.909180 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:59.909183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:59.909294 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 06:22:59.909370 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:22:59.909582 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":114,"threads_started":1}
I20260928 06:22:59.910393 2060 heartbeater.cc:347] Connected to a master server at 127.31.214.126:33903
I20260928 06:22:59.910493 2060 heartbeater.cc:464] Registering TS with master...
I20260928 06:22:59.910692 2060 heartbeater.cc:511] Master 127.31.214.126:33903 requested a full tablet report, sending...
I20260928 06:22:59.911278 1856 ts_manager.cc:194] Registered new tserver with Master: b5349192348940f99ca4e6f57033b584 (127.31.214.65:43391)
I20260928 06:22:59.912256 1856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:58995
I20260928 06:22:59.914513 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 1929
I20260928 06:22:59.914629 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:22:59.914819 32601 external_mini_cluster.cc:1428] Setting key 6c437b4c6d3413f8392ed0b9d8c626fd
I20260928 06:22:59.917348 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:59.911473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:33903 (local address 127.31.214.65:47751)
0928 06:22:59.911643 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:22:59.911654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:22:59.911716 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:22:59.911832 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:22:59.911833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:22:59.912020 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:22:59.912083 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:59.912084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.913001 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.913002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:59.913517 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:22:59.913519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.913858 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.913859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:22:59.913916 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:59.914382 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:22:59.914386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:22:59.914657 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:22:59.916272 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:59.916273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:59.916273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:59.916451 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:59.916675 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:22:59.916676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:22:59.916676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:22:59.916741 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:22:59.917005 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:22:59.917006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:22:59.917061 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 06:22:59.917147 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:22:59.917252 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260928 06:22:59.917341 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:22:59.911544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:47751 (local address 127.31.214.126:33903)
0928 06:22:59.911581 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:22:59.911583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:22:59.911720 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:22:59.911771 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:22:59.911772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:22:59.911780 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:22:59.911820 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:22:59.912169 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.912899 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.913602 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:22:59.913789 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:22:59.913848 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:22:59.914747 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:22:59.914748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:22:59.914748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:22:59.914756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:22:59.916151 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:59.916547 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:59.916548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:59.916548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:59.916607 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:22:59.916825 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:22:59.916826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:22:59.916826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:22:59.916938 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:22:59.917013 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:22:59.917174 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:22:59.917248 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:22:59.945075 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:33903
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:00.039603 2069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:00.039829 2069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:00.039855 2069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:00.040984 2069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:00.041023 2069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:00.041044 2069 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:23:00.041056 2069 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:23:00.041074 2069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:00.042614 2069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.214.126:33903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:00.042821 2069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:00.043015 2069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:00.048283 2077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:00.048237 2075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:00.048237 2074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:00.048838 2069 server_base.cc:1034] running on GCE node
I20260928 06:23:00.049134 2069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:00.049448 2069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:00.050714 2069 hybrid_clock.cc:648] HybridClock initialized: now 1790576580050680 us; error 49 us; skew 500 ppm
Sep 28 06:23:00 dist-test-slave-q03f krb5kdc[1812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576580, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:00.053489 2069 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:23:00.054118 2069 webserver.cc:460] Webserver started at http://127.31.214.66:33423/ using document root <none> and password file <none>
I20260928 06:23:00.054364 2069 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:00.054435 2069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:00.054585 2069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:00.055680 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "890c8688de7f4a2f998c2520df9a3a17"
format_stamp: "Formatted at 2026-09-28 06:23:00 on dist-test-slave-q03f"
server_key: "48cb1f0ee78065d9fa3b5a627b3d7c72"
server_key_iv: "efb4afba1b07b3bf50c35dfcea25fbb4"
server_key_version: "encryptionkey@0"
I20260928 06:23:00.056066 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "890c8688de7f4a2f998c2520df9a3a17"
format_stamp: "Formatted at 2026-09-28 06:23:00 on dist-test-slave-q03f"
server_key: "48cb1f0ee78065d9fa3b5a627b3d7c72"
server_key_iv: "efb4afba1b07b3bf50c35dfcea25fbb4"
server_key_version: "encryptionkey@0"
I20260928 06:23:00.057538 2069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:00.058833 2084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:00.059159 2069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:00.059252 2069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "890c8688de7f4a2f998c2520df9a3a17"
format_stamp: "Formatted at 2026-09-28 06:23:00 on dist-test-slave-q03f"
server_key: "48cb1f0ee78065d9fa3b5a627b3d7c72"
server_key_iv: "efb4afba1b07b3bf50c35dfcea25fbb4"
server_key_version: "encryptionkey@0"
I20260928 06:23:00.059310 2069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:00.071329 2069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:00.073727 2069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:00.073846 2069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:00.074393 2069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:00.074434 2069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:00.074465 2069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:00.074525 2069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:00.082010 2069 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:34531
I20260928 06:23:00.082093 2199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:34531 every 8 connection(s)
I20260928 06:23:00.082463 2069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:23:00 dist-test-slave-q03f krb5kdc[1812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576580, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:00.086344 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.075119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:38829 (local address 127.31.214.126:33903)
0928 06:23:00.075173 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.075175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.075381 (+ 206us) server_negotiation.cc:407] Connection header received
0928 06:23:00.075969 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.075970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.076012 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.076081 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:00.076844 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.077757 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.079595 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.079930 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.080010 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.083335 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:00.083336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:00.083336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:00.083344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:00.084893 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.085351 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.085352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.085352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.085410 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.085677 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.085678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.085679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.085798 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:00.085852 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.086198 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.086275 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:23:00.086622 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.075075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:33903 (local address 127.31.214.66:38829)
0928 06:23:00.075251 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.075288 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.075838 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.076177 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.076188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.076503 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:00.076738 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.076743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.077868 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.077869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.079481 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.079485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.080029 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.080030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.080207 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.081180 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:00.081193 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:00.083216 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:00.084986 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.084993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.084995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.085258 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.085486 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.085487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.085487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.085610 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.085861 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:00.085866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:00.086057 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.086158 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.086299 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260928 06:23:00.087193 2200 heartbeater.cc:347] Connected to a master server at 127.31.214.126:33903
I20260928 06:23:00.087275 2200 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:00.087488 2200 heartbeater.cc:511] Master 127.31.214.126:33903 requested a full tablet report, sending...
I20260928 06:23:00.087966 1857 ts_manager.cc:194] Registered new tserver with Master: 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531)
I20260928 06:23:00.088424 1857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:38829
I20260928 06:23:00.090605 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 2069
I20260928 06:23:00.090685 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:00.090903 32601 external_mini_cluster.cc:1428] Setting key 62e13524cdaa4ff3d011704851175658
I20260928 06:23:00.095105 2206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.088438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:33903 (local address 127.31.214.66:53009)
0928 06:23:00.088728 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.088738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.088844 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.088978 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.088979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.089156 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:00.089207 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.089209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.089865 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.089866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.090443 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.090444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.090907 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.090908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.090985 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.091556 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:00.091636 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:00.091980 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:00.093879 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.093880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.093881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.094155 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.094361 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.094362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.094362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.094431 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.094810 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:00.094811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:00.094859 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.094946 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.095032 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":213,"threads_started":1}
I20260928 06:23:00.095268 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.088484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:53009 (local address 127.31.214.126:33903)
0928 06:23:00.088543 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.088546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.088831 (+ 285us) server_negotiation.cc:407] Connection header received
0928 06:23:00.088909 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.088910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.088921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.088965 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:00.089281 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.089762 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.090538 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.090817 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.090896 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.092079 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:00.092081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:00.092081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:00.092088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:00.093779 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.094237 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.094238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.094239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.094292 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.094529 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.094529 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.094530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.094737 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:00.094810 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.094999 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.095125 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:23:00.120976 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:33903
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:33415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:00.207401 2209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:00.207619 2209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:00.207652 2209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:00.208737 2209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:00.208775 2209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:00.208796 2209 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:23:00.208807 2209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:23:00.208824 2209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:00.210592 2209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:33415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.214.126:33903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:00.210891 2209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:00.211174 2209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:00.216678 2217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:00.216820 2209 server_base.cc:1034] running on GCE node
W20260928 06:23:00.216678 2215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:00.216970 2214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:00.217168 2209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:00.217371 2209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:00.218514 2209 hybrid_clock.cc:648] HybridClock initialized: now 1790576580218489 us; error 40 us; skew 500 ppm
Sep 28 06:23:00 dist-test-slave-q03f krb5kdc[1812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576580, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:00.221112 2209 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:23:00.221643 2209 webserver.cc:460] Webserver started at http://127.31.214.67:38141/ using document root <none> and password file <none>
I20260928 06:23:00.221846 2209 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:00.221890 2209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:00.222069 2209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:00.223351 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "672ee895b5454350b7cfee7aa77f3cc6"
format_stamp: "Formatted at 2026-09-28 06:23:00 on dist-test-slave-q03f"
server_key: "75e0bf8ce66091e713302d13a6693527"
server_key_iv: "341c6801094443149067ded99f2ed978"
server_key_version: "encryptionkey@0"
I20260928 06:23:00.223676 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "672ee895b5454350b7cfee7aa77f3cc6"
format_stamp: "Formatted at 2026-09-28 06:23:00 on dist-test-slave-q03f"
server_key: "75e0bf8ce66091e713302d13a6693527"
server_key_iv: "341c6801094443149067ded99f2ed978"
server_key_version: "encryptionkey@0"
I20260928 06:23:00.225028 2209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:00.225839 2224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:00.226215 2209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:00.226322 2209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "672ee895b5454350b7cfee7aa77f3cc6"
format_stamp: "Formatted at 2026-09-28 06:23:00 on dist-test-slave-q03f"
server_key: "75e0bf8ce66091e713302d13a6693527"
server_key_iv: "341c6801094443149067ded99f2ed978"
server_key_version: "encryptionkey@0"
I20260928 06:23:00.226482 2209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:00.244095 2209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:00.246773 2209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:00.246974 2209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:00.247640 2209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:00.247709 2209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:00.247748 2209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:00.247777 2209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:23:00 dist-test-slave-q03f krb5kdc[1812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576580, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:00.255550 2209 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:35817
I20260928 06:23:00.255956 2209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:00.256033 2339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:35817 every 8 connection(s)
I20260928 06:23:00.256731 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 2209
I20260928 06:23:00.256824 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:00.257027 32601 external_mini_cluster.cc:1428] Setting key 5fca95a6cc4abbcd391a07398c431f0d
I20260928 06:23:00.258787 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.248273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:52049 (local address 127.31.214.126:33903)
0928 06:23:00.248329 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.248331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.248571 (+ 240us) server_negotiation.cc:407] Connection header received
0928 06:23:00.249187 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.249189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.249210 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.249268 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:00.250054 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.250794 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.251930 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.252166 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.252255 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.255157 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:00.255159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:00.255159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:00.255166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:00.257023 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.257701 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.257703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.257703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.257771 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.258157 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.258158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.258158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.258298 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:00.258353 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.258652 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.258723 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:23:00.258957 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.248264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:33903 (local address 127.31.214.67:52049)
0928 06:23:00.248434 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.248473 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.249021 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.249352 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.249355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.249681 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:00.249897 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.249902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.250898 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.250899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.251822 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.251825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.252262 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.252263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.252452 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.253039 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:00.253058 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:00.255029 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:00.257118 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.257124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.257126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.257476 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.257844 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.257845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.257846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.258084 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.258358 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:00.258363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:00.258489 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.258570 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.258698 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260928 06:23:00.259441 2340 heartbeater.cc:347] Connected to a master server at 127.31.214.126:33903
I20260928 06:23:00.259539 2340 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:00.259717 2340 heartbeater.cc:511] Master 127.31.214.126:33903 requested a full tablet report, sending...
I20260928 06:23:00.260486 1856 ts_manager.cc:194] Registered new tserver with Master: 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:00.261052 1856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:52049
I20260928 06:23:00.266799 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.260476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:46235 (local address 127.31.214.126:33903)
0928 06:23:00.260536 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.260539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.260688 (+ 149us) server_negotiation.cc:407] Connection header received
0928 06:23:00.260748 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.260749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.260758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.260808 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:00.261155 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.261992 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.262759 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.262959 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.263022 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.264120 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:00.264121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:00.264121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:00.264127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:00.265445 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.265815 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.265816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.265817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.265996 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.266236 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.266237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.266237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.266352 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:00.266417 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.266664 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.266734 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:23:00.266815 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.260428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:33903 (local address 127.31.214.67:46235)
0928 06:23:00.260619 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.260630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.260693 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.260812 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.260812 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.261002 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:00.261080 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.261081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.262083 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.262083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.262671 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.262673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.263031 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.263032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.263123 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.263725 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:00.263729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:00.264031 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:00.265530 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.265531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.265531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.265724 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.266087 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.266088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.266088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.266157 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.266405 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:00.266406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:00.266509 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.266575 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.266721 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":129,"threads_started":1}
I20260928 06:23:00.269112 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:23:00 dist-test-slave-q03f krb5kdc[1812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:23:00 dist-test-slave-q03f krb5kdc[1812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:00.293998 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.286363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.31.214.126:33903)
0928 06:23:00.286441 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.286443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.286506 (+ 63us) server_negotiation.cc:407] Connection header received
0928 06:23:00.286591 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.286592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.286610 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.286673 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:00.287347 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.287862 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.288717 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.288925 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.289035 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.291451 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:00.291453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:00.291453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:00.291461 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:00.292791 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.293168 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.293169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.293169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.293227 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.293424 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.293424 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.293425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.293549 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:00.293655 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.293777 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.293862 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:23:00.295814 1856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:00.297405 1856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:00.304637 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.300527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:35817 (local address 127.0.0.1:41704)
0928 06:23:00.300665 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.300677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.300721 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.301199 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.301200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.301201 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:00.301253 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.301255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.302137 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.302138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.303577 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.303579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.304302 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.304303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.304386 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.304400 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.304470 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.304539 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":37,"thread_start_us":93,"threads_started":1}
I20260928 06:23:00.304773 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.300299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:43391 (local address 127.0.0.1:52010)
0928 06:23:00.300480 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.300491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.300539 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.301236 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.301237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.301254 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:00.301302 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.301303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.302538 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.302539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.303468 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.303471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.304445 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.304446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.304511 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.304514 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.304573 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.304605 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":132,"threads_started":1}
I20260928 06:23:00.304888 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.300057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:34531 (local address 127.0.0.1:36324)
0928 06:23:00.300181 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.300227 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.300292 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.300605 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.300608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.300610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:00.300892 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.300894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.302220 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.302220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.303576 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.303579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.304668 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.304669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.304727 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.304730 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.304783 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.304823 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":75,"thread_start_us":74,"threads_started":1}
I20260928 06:23:00.305001 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.300720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.31.214.67:35817)
0928 06:23:00.301035 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.301039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.301049 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:23:00.301085 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.301086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.301103 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.301202 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:00.301309 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.302046 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.303684 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.304206 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.304301 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.304365 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.304865 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.304916 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":92,"threads_started":1}
I20260928 06:23:00.305142 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.300183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.31.214.66:34531)
0928 06:23:00.300348 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.300354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.300366 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:23:00.300411 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.300413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.300431 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.300537 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:00.300963 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.302150 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.303668 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.304370 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.304449 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.304506 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.304976 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.305039 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260928 06:23:00.305688 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.300844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.31.214.65:43391)
0928 06:23:00.301063 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.301067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.301077 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:23:00.301115 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.301117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.301131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.301388 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:00.301446 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.302452 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.303571 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.304370 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.304449 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.304506 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.304595 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.304629 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
I20260928 06:23:00.305691 2274 tablet_service.cc:1467] Processing CreateTablet for tablet d115c3eca69f475795767c8da7e3bec9 (DEFAULT_TABLE table=test-table [id=10d45d1a7cd14cd99e123efb9fc4ea47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:00.305874 2134 tablet_service.cc:1467] Processing CreateTablet for tablet d115c3eca69f475795767c8da7e3bec9 (DEFAULT_TABLE table=test-table [id=10d45d1a7cd14cd99e123efb9fc4ea47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:00.306155 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d115c3eca69f475795767c8da7e3bec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:00.306250 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d115c3eca69f475795767c8da7e3bec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:00.306398 1994 tablet_service.cc:1467] Processing CreateTablet for tablet d115c3eca69f475795767c8da7e3bec9 (DEFAULT_TABLE table=test-table [id=10d45d1a7cd14cd99e123efb9fc4ea47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:00.306735 1994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d115c3eca69f475795767c8da7e3bec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:00.309751 2363 tablet_bootstrap.cc:492] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: Bootstrap starting.
I20260928 06:23:00.310098 2365 tablet_bootstrap.cc:492] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: Bootstrap starting.
I20260928 06:23:00.310714 2365 tablet_bootstrap.cc:654] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:00.310833 2363 tablet_bootstrap.cc:654] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:00.311147 2365 log.cc:824] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:00.311160 2363 log.cc:824] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:00.311913 2365 tablet_bootstrap.cc:492] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: No bootstrap required, opened a new log
I20260928 06:23:00.311968 2365 ts_tablet_manager.cc:1397] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:00.312041 2363 tablet_bootstrap.cc:492] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: No bootstrap required, opened a new log
I20260928 06:23:00.312131 2363 ts_tablet_manager.cc:1397] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:23:00.313385 2365 raft_consensus.cc:348] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.313570 2365 raft_consensus.cc:374] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:00.313644 2365 raft_consensus.cc:729] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890c8688de7f4a2f998c2520df9a3a17, State: Initialized, Role: FOLLOWER
I20260928 06:23:00.313509 2363 raft_consensus.cc:348] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.313694 2363 raft_consensus.cc:374] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:00.313735 2363 raft_consensus.cc:729] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672ee895b5454350b7cfee7aa77f3cc6, State: Initialized, Role: FOLLOWER
I20260928 06:23:00.313793 2365 consensus_queue.cc:260] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.313849 2363 consensus_queue.cc:260] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.314147 2365 ts_tablet_manager.cc:1428] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:23:00.314180 2340 heartbeater.cc:503] Master 127.31.214.126:33903 was elected leader, sending a full tablet report...
I20260928 06:23:00.314162 2200 heartbeater.cc:503] Master 127.31.214.126:33903 was elected leader, sending a full tablet report...
I20260928 06:23:00.314218 2363 ts_tablet_manager.cc:1428] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:23:00.311654 2364 tablet_bootstrap.cc:492] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: Bootstrap starting.
I20260928 06:23:00.314970 2364 tablet_bootstrap.cc:654] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:00.315359 2364 log.cc:824] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:00.316149 2364 tablet_bootstrap.cc:492] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: No bootstrap required, opened a new log
I20260928 06:23:00.316207 2364 ts_tablet_manager.cc:1397] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:23:00.317633 2364 raft_consensus.cc:348] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.317765 2364 raft_consensus.cc:374] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:00.317787 2364 raft_consensus.cc:729] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5349192348940f99ca4e6f57033b584, State: Initialized, Role: FOLLOWER
I20260928 06:23:00.317934 2364 consensus_queue.cc:260] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.318274 2364 ts_tablet_manager.cc:1428] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:23:00.318341 2060 heartbeater.cc:503] Master 127.31.214.126:33903 was elected leader, sending a full tablet report...
W20260928 06:23:00.333629 2201 tablet.cc:2369] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:23:00.408464 2061 tablet.cc:2369] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:23:00.503252 2371 raft_consensus.cc:482] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:00.503429 2371 raft_consensus.cc:504] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.503850 2371 leader_election.cc:288] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:00.507068 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.504407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:60499 (local address 127.31.214.67:35817)
0928 06:23:00.504488 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.504491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.504515 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:23:00.504574 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.504575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.504584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.504625 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:00.504790 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.505300 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.506322 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.506702 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.506788 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.506798 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.506956 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.506994 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:23:00.507081 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.504267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:35817 (local address 127.31.214.65:60499)
0928 06:23:00.504449 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.504461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.504513 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.504639 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.504640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.504642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:00.504718 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.504719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.505408 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.505410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.506215 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.506217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.506789 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.506789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.506867 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.506870 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.506947 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.506991 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":60,"threads_started":1}
I20260928 06:23:00.507372 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.504164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:34123 (local address 127.31.214.66:34531)
0928 06:23:00.504248 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.504251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.504380 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:23:00.504554 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.504555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.504574 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.504654 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:00.504832 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.505382 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.506511 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.506903 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.507014 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.507022 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.507221 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.507297 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 06:23:00.507421 2341 tablet.cc:2369] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:23:00.507647 2372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.504082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:34531 (local address 127.31.214.65:34123)
0928 06:23:00.504242 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.504260 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.504388 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.504658 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.504659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.504662 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:00.504752 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.504754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.505557 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.505558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.506414 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.506416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.507014 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.507015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.507098 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.507102 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.507515 (+ 413us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.507566 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260928 06:23:00.507812 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d115c3eca69f475795767c8da7e3bec9" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6" is_pre_election: true
I20260928 06:23:00.507995 2294 raft_consensus.cc:2454] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 0.
I20260928 06:23:00.508056 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d115c3eca69f475795767c8da7e3bec9" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890c8688de7f4a2f998c2520df9a3a17" is_pre_election: true
I20260928 06:23:00.508245 2154 raft_consensus.cc:2454] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 0.
I20260928 06:23:00.508309 1945 leader_election.cc:302] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672ee895b5454350b7cfee7aa77f3cc6, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:00.508451 2371 raft_consensus.cc:2781] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:00.508528 2371 raft_consensus.cc:482] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:00.508546 2371 raft_consensus.cc:3037] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:00.509333 2371 raft_consensus.cc:504] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.509457 2371 leader_election.cc:288] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 election: Requested vote from peers 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:00.509688 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d115c3eca69f475795767c8da7e3bec9" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890c8688de7f4a2f998c2520df9a3a17"
I20260928 06:23:00.509711 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d115c3eca69f475795767c8da7e3bec9" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6"
I20260928 06:23:00.509776 2294 raft_consensus.cc:3037] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:00.509770 2154 raft_consensus.cc:3037] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:00.510761 2294 raft_consensus.cc:2454] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 1.
I20260928 06:23:00.510761 2154 raft_consensus.cc:2454] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 1.
I20260928 06:23:00.511005 1945 leader_election.cc:302] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890c8688de7f4a2f998c2520df9a3a17, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:00.511152 2371 raft_consensus.cc:2781] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:00.511273 2371 raft_consensus.cc:686] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 1 LEADER]: Becoming Leader. State: Replica: b5349192348940f99ca4e6f57033b584, State: Running, Role: LEADER
I20260928 06:23:00.511420 2371 consensus_queue.cc:237] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.512326 1857 catalog_manager.cc:5495] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5349192348940f99ca4e6f57033b584 (127.31.214.65). New cstate: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:23:00 dist-test-slave-q03f krb5kdc[1812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576579, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:00.554826 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.547026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.31.214.126:33903)
0928 06:23:00.547133 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.547136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.547274 (+ 138us) server_negotiation.cc:407] Connection header received
0928 06:23:00.547380 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.547381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.547393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.547448 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:00.548008 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.548610 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.549412 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.549639 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.549736 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.552090 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:00.552091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:00.552092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:00.552101 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:00.553398 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.553830 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.553831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.553832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.553895 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:00.554147 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:00.554148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:00.554149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:00.554274 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:00.554355 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.554544 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.554695 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":73}
I20260928 06:23:00.554916 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.546877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:33903 (local address 127.0.0.1:52838)
0928 06:23:00.547171 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.547190 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.547289 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.547464 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.547465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.547825 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:00.547884 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.547887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.548714 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.548715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.549314 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.549316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.549738 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.549738 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.549806 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.550486 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:00.550491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:00.551988 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:00.553502 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.553511 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.553513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.553726 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.554007 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:00.554008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:00.554008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:00.554076 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:00.554356 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:00.554366 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:00.554432 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.554514 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.554761 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":124,"thread_start_us":140,"threads_started":1}
I20260928 06:23:00.557314 1857 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:52838:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:23:00.557518 1857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:00.559473 2134 tablet_service.cc:1467] Processing CreateTablet for tablet f670eeba2c3b49c281d3c72b50d45533 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dbeab394f14e4adf8a176b4020747ff2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:23:00.559463 1994 tablet_service.cc:1467] Processing CreateTablet for tablet f670eeba2c3b49c281d3c72b50d45533 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dbeab394f14e4adf8a176b4020747ff2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:23:00.559499 2274 tablet_service.cc:1467] Processing CreateTablet for tablet f670eeba2c3b49c281d3c72b50d45533 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dbeab394f14e4adf8a176b4020747ff2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:23:00.559662 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f670eeba2c3b49c281d3c72b50d45533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:00.559669 1994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f670eeba2c3b49c281d3c72b50d45533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:00.559669 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f670eeba2c3b49c281d3c72b50d45533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:00.561158 2365 tablet_bootstrap.cc:492] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: Bootstrap starting.
I20260928 06:23:00.561388 2364 tablet_bootstrap.cc:492] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: Bootstrap starting.
I20260928 06:23:00.561395 2363 tablet_bootstrap.cc:492] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: Bootstrap starting.
I20260928 06:23:00.561916 2364 tablet_bootstrap.cc:654] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:00.561918 2365 tablet_bootstrap.cc:654] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:00.562117 2363 tablet_bootstrap.cc:654] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:00.562892 2363 tablet_bootstrap.cc:492] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: No bootstrap required, opened a new log
I20260928 06:23:00.562976 2365 tablet_bootstrap.cc:492] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: No bootstrap required, opened a new log
I20260928 06:23:00.562968 2363 ts_tablet_manager.cc:1397] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:23:00.563016 2365 ts_tablet_manager.cc:1397] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:00.563093 2364 tablet_bootstrap.cc:492] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: No bootstrap required, opened a new log
I20260928 06:23:00.563153 2364 ts_tablet_manager.cc:1397] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:00.563180 2363 raft_consensus.cc:348] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.563205 2365 raft_consensus.cc:348] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.563273 2363 raft_consensus.cc:374] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:00.563283 2365 raft_consensus.cc:374] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:00.563313 2363 raft_consensus.cc:729] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672ee895b5454350b7cfee7aa77f3cc6, State: Initialized, Role: FOLLOWER
I20260928 06:23:00.563313 2365 raft_consensus.cc:729] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890c8688de7f4a2f998c2520df9a3a17, State: Initialized, Role: FOLLOWER
I20260928 06:23:00.563356 2365 consensus_queue.cc:260] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.563374 2363 consensus_queue.cc:260] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.563484 2365 ts_tablet_manager.cc:1428] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:00.563490 2363 ts_tablet_manager.cc:1428] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:23:00.563552 2364 raft_consensus.cc:348] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.563639 2364 raft_consensus.cc:374] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:00.563670 2364 raft_consensus.cc:729] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5349192348940f99ca4e6f57033b584, State: Initialized, Role: FOLLOWER
I20260928 06:23:00.563637 2368 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.563719 2368 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:00.563730 2364 consensus_queue.cc:260] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.563853 2364 ts_tablet_manager.cc:1428] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:23:00.563908 2371 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.564013 2371 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:00.564018 2369 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.564136 2369 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:00.784646 2371 raft_consensus.cc:482] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:00.784757 2371 raft_consensus.cc:504] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.784981 2371 leader_election.cc:288] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:00.785357 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f670eeba2c3b49c281d3c72b50d45533" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6" is_pre_election: true
I20260928 06:23:00.785368 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f670eeba2c3b49c281d3c72b50d45533" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890c8688de7f4a2f998c2520df9a3a17" is_pre_election: true
I20260928 06:23:00.785488 2294 raft_consensus.cc:2454] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 0.
I20260928 06:23:00.785487 2154 raft_consensus.cc:2454] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 0.
I20260928 06:23:00.785707 1945 leader_election.cc:302] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890c8688de7f4a2f998c2520df9a3a17, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:00.785845 2371 raft_consensus.cc:2781] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:00.785980 2371 raft_consensus.cc:482] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:00.786013 2371 raft_consensus.cc:3037] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:00.786823 2371 raft_consensus.cc:504] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.786984 2371 leader_election.cc:288] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 election: Requested vote from peers 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:00.787241 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f670eeba2c3b49c281d3c72b50d45533" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890c8688de7f4a2f998c2520df9a3a17"
I20260928 06:23:00.787334 2154 raft_consensus.cc:3037] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:00.787281 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f670eeba2c3b49c281d3c72b50d45533" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6"
I20260928 06:23:00.787371 2294 raft_consensus.cc:3037] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:00.788033 2154 raft_consensus.cc:2454] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 1.
I20260928 06:23:00.788051 2294 raft_consensus.cc:2454] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 1.
I20260928 06:23:00.788223 1945 leader_election.cc:302] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890c8688de7f4a2f998c2520df9a3a17, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:00.788408 2371 raft_consensus.cc:2781] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:00.788476 2371 raft_consensus.cc:686] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [term 1 LEADER]: Becoming Leader. State: Replica: b5349192348940f99ca4e6f57033b584, State: Running, Role: LEADER
I20260928 06:23:00.788534 2371 consensus_queue.cc:237] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.788903 2374 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: TxnStatusTablet state changed. Reason: New leader b5349192348940f99ca4e6f57033b584. Latest consensus state: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.789014 2374 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:23:00.789248 2386 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:23:00.789230 1857 catalog_manager.cc:5495] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5349192348940f99ca4e6f57033b584 (127.31.214.65). New cstate: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:00.789387 2386 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:23:00.850980 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.847809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.31.214.65:43391)
0928 06:23:00.847947 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.847948 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.847954 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:23:00.847995 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.847996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.848044 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.848122 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:00.848317 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.848884 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.850137 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.850333 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.850390 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.850627 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:00.850730 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:00.850790 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.850888 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.850906 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260928 06:23:00.851027 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.847723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:43391 (local address 127.0.0.1:52018)
0928 06:23:00.847814 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.847824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.847861 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.848162 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.848164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.848166 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:23:00.848227 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.848229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.848985 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.848987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.850048 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.850051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.850425 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.850426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.850502 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.850524 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:23:00.850805 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:23:00.850816 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.850881 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.850938 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 06:23:00.853052 2294 raft_consensus.cc:1261] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Refusing update from remote peer b5349192348940f99ca4e6f57033b584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:00.853058 2154 raft_consensus.cc:1261] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Refusing update from remote peer b5349192348940f99ca4e6f57033b584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:00.853565 2374 consensus_queue.cc:1035] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:00.853729 2371 consensus_queue.cc:1035] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:00.855100 2369 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: TxnStatusTablet state changed. Reason: New leader b5349192348940f99ca4e6f57033b584. Latest consensus state: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.855194 2369 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:00.855109 2368 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: TxnStatusTablet state changed. Reason: New leader b5349192348940f99ca4e6f57033b584. Latest consensus state: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.855235 2368 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:00.856585 2374 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.856683 2374 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:23:00.856720 2368 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.856812 2368 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P 890c8688de7f4a2f998c2520df9a3a17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:00.857122 2371 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.857216 2371 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P b5349192348940f99ca4e6f57033b584: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:23:00.857481 2391 mvcc.cc:205] Tried to move back new op lower bound from 7334201675170869248 to 7334201674909974528. Current Snapshot: MvccSnapshot[applied={T|T < 7334201675170869248}]
I20260928 06:23:00.857854 2369 tablet_replica.cc:439] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } } }
I20260928 06:23:00.858237 2369 tablet_replica.cc:442] T f670eeba2c3b49c281d3c72b50d45533 P 672ee895b5454350b7cfee7aa77f3cc6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:00.858460 2392 mvcc.cc:205] Tried to move back new op lower bound from 7334201675170869248 to 7334201674909974528. Current Snapshot: MvccSnapshot[applied={T|T < 7334201675170869248}]
I20260928 06:23:00.861915 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.859225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.31.214.65:43391)
0928 06:23:00.859288 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.859290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.859419 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:23:00.859487 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.859488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.859499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.859549 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:00.859722 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.860277 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.861192 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.861406 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.861485 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.861643 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:00.861685 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:00.861742 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.861801 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.861825 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:23:00.867470 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.864920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:42823 (local address 127.31.214.65:43391)
0928 06:23:00.865096 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:00.865098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:00.865110 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:23:00.865176 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:00.865177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:00.865187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:00.865241 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:00.865395 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.865892 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.866738 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.866920 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.867001 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.867190 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:00.867227 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:00.867288 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:00.867346 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:23:00.867371 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":156}
I20260928 06:23:00.867739 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:00.864762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:43391 (local address 127.31.214.65:42823)
0928 06:23:00.864920 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:00.864932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:00.865127 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:00.865244 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:00.865244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:00.865245 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:23:00.865321 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.865323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.866004 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.866005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.866643 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:00.866645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:00.867002 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:00.867003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:00.867076 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:00.867089 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:23:00.867287 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:23:00.867288 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:23:00.867342 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:23:00.867392 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":8,"thread_start_us":85,"threads_started":1}
I20260928 06:23:00.870391 2294 raft_consensus.cc:1261] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Refusing update from remote peer b5349192348940f99ca4e6f57033b584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:00.870417 2154 raft_consensus.cc:1261] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Refusing update from remote peer b5349192348940f99ca4e6f57033b584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:00.870637 2374 consensus_queue.cc:1035] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:00.870810 2403 consensus_queue.cc:1035] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:00.872000 2390 mvcc.cc:205] Tried to move back new op lower bound from 7334201675243159552 to 7334201673775292416. Current Snapshot: MvccSnapshot[applied={T|T < 7334201675243159552}]
I20260928 06:23:00.886320 1857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:00.886478 1857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:00.887377 1857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9: Sending DeleteTablet for 3 replicas of tablet d115c3eca69f475795767c8da7e3bec9
I20260928 06:23:00.887786 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet d115c3eca69f475795767c8da7e3bec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:00 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:41704
I20260928 06:23:00.887871 2134 tablet_service.cc:1514] Processing DeleteTablet for tablet d115c3eca69f475795767c8da7e3bec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:00 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:36324
I20260928 06:23:00.888053 1857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:00.888252 1857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:00.888325 2408 tablet_replica.cc:330] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: stopping tablet replica
I20260928 06:23:00.888463 2408 raft_consensus.cc:2229] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:00.888422 1993 tablet_service.cc:1514] Processing DeleteTablet for tablet d115c3eca69f475795767c8da7e3bec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:00 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:52010
I20260928 06:23:00.888655 2408 raft_consensus.cc:2258] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:00.888835 2407 tablet_replica.cc:330] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: stopping tablet replica
I20260928 06:23:00.889025 2407 raft_consensus.cc:2229] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:00.889153 2407 raft_consensus.cc:2258] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:00.889367 2408 ts_tablet_manager.cc:1905] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:00.890080 1993 tablet_service.cc:1467] Processing CreateTablet for tablet 9858605c89ae4f5c9e8c36030f4de57c (DEFAULT_TABLE table=test-table [id=6db591c75711454ab200a96fb2e501a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:00.890228 1993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9858605c89ae4f5c9e8c36030f4de57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:00.890232 2407 ts_tablet_manager.cc:1905] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:00.891809 2408 ts_tablet_manager.cc:1918] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:00.891927 2408 log.cc:1196] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/d115c3eca69f475795767c8da7e3bec9
I20260928 06:23:00.892057 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 9858605c89ae4f5c9e8c36030f4de57c (DEFAULT_TABLE table=test-table [id=6db591c75711454ab200a96fb2e501a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:00.892182 2407 ts_tablet_manager.cc:1918] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:00.892227 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9858605c89ae4f5c9e8c36030f4de57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:00.892269 2407 log.cc:1196] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/d115c3eca69f475795767c8da7e3bec9
I20260928 06:23:00.892342 2408 ts_tablet_manager.cc:1939] T d115c3eca69f475795767c8da7e3bec9 P 890c8688de7f4a2f998c2520df9a3a17: Deleting consensus metadata
I20260928 06:23:00.892558 2407 ts_tablet_manager.cc:1939] T d115c3eca69f475795767c8da7e3bec9 P 672ee895b5454350b7cfee7aa77f3cc6: Deleting consensus metadata
I20260928 06:23:00.893558 2364 tablet_bootstrap.cc:492] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: Bootstrap starting.
I20260928 06:23:00.893569 2365 tablet_bootstrap.cc:492] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: Bootstrap starting.
I20260928 06:23:00.894047 2409 tablet_replica.cc:330] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: stopping tablet replica
I20260928 06:23:00.894048 2364 tablet_bootstrap.cc:654] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:00.894121 2365 tablet_bootstrap.cc:654] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:00.894193 2409 raft_consensus.cc:2229] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:23:00.894301 2409 raft_consensus.cc:2258] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:00.894575 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 9858605c89ae4f5c9e8c36030f4de57c (DEFAULT_TABLE table=test-table [id=6db591c75711454ab200a96fb2e501a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:00.894699 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9858605c89ae4f5c9e8c36030f4de57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:00.896057 2363 tablet_bootstrap.cc:492] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: Bootstrap starting.
I20260928 06:23:00.896313 1841 catalog_manager.cc:4841] TS 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531): tablet d115c3eca69f475795767c8da7e3bec9 (table test-table [id=10d45d1a7cd14cd99e123efb9fc4ea47]) successfully deleted
I20260928 06:23:00.896482 1841 catalog_manager.cc:4841] TS 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817): tablet d115c3eca69f475795767c8da7e3bec9 (table test-table [id=10d45d1a7cd14cd99e123efb9fc4ea47]) successfully deleted
I20260928 06:23:00.896634 2363 tablet_bootstrap.cc:654] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:00.897246 2409 ts_tablet_manager.cc:1905] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:00.897341 2363 tablet_bootstrap.cc:492] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: No bootstrap required, opened a new log
I20260928 06:23:00.897400 2363 ts_tablet_manager.cc:1397] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:00.897569 2363 raft_consensus.cc:348] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.897655 2363 raft_consensus.cc:374] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:00.897686 2363 raft_consensus.cc:729] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672ee895b5454350b7cfee7aa77f3cc6, State: Initialized, Role: FOLLOWER
I20260928 06:23:00.897730 2363 consensus_queue.cc:260] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.897861 2363 ts_tablet_manager.cc:1428] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:00.899344 2409 ts_tablet_manager.cc:1918] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:00.899434 2409 log.cc:1196] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/d115c3eca69f475795767c8da7e3bec9
I20260928 06:23:00.899794 2409 ts_tablet_manager.cc:1939] T d115c3eca69f475795767c8da7e3bec9 P b5349192348940f99ca4e6f57033b584: Deleting consensus metadata
I20260928 06:23:00.900282 1841 catalog_manager.cc:4841] TS b5349192348940f99ca4e6f57033b584 (127.31.214.65:43391): tablet d115c3eca69f475795767c8da7e3bec9 (table test-table [id=10d45d1a7cd14cd99e123efb9fc4ea47]) successfully deleted
I20260928 06:23:00.900527 2364 tablet_bootstrap.cc:492] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: No bootstrap required, opened a new log
I20260928 06:23:00.900589 2364 ts_tablet_manager.cc:1397] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260928 06:23:00.900739 2364 raft_consensus.cc:348] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.900825 2364 raft_consensus.cc:374] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:00.900847 2364 raft_consensus.cc:729] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5349192348940f99ca4e6f57033b584, State: Initialized, Role: FOLLOWER
I20260928 06:23:00.900893 2364 consensus_queue.cc:260] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.901000 2364 ts_tablet_manager.cc:1428] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:23:00.901794 2365 tablet_bootstrap.cc:492] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: No bootstrap required, opened a new log
I20260928 06:23:00.901866 2365 ts_tablet_manager.cc:1397] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260928 06:23:00.902060 2365 raft_consensus.cc:348] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.902144 2365 raft_consensus.cc:374] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:00.902168 2365 raft_consensus.cc:729] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890c8688de7f4a2f998c2520df9a3a17, State: Initialized, Role: FOLLOWER
I20260928 06:23:00.902222 2365 consensus_queue.cc:260] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:00.902356 2365 ts_tablet_manager.cc:1428] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:01.000015 2368 raft_consensus.cc:482] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:01.000137 2368 raft_consensus.cc:504] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:01.000535 2368 leader_election.cc:288] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5349192348940f99ca4e6f57033b584 (127.31.214.65:43391), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:01.004796 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:01.000728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:43391 (local address 127.31.214.66:45337)
0928 06:23:01.000939 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:01.000952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:01.001019 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:01.001211 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:01.001212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:01.001214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:01.001276 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:01.001278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.002217 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.002219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:01.003457 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:01.003461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.004485 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.004486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:01.004550 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:01.004555 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:23:01.004626 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:23:01.004678 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":157,"threads_started":1}
I20260928 06:23:01.005337 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:01.000800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:45337 (local address 127.31.214.65:43391)
0928 06:23:01.000873 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:01.000876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:01.001025 (+ 149us) server_negotiation.cc:407] Connection header received
0928 06:23:01.001093 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:01.001095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:01.001106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:01.001137 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:01.001352 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.002093 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.003573 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.004343 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.004430 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:01.004442 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:01.005174 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 06:23:01.005238 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:23:01.005599 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:01.001137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:36383 (local address 127.31.214.67:35817)
0928 06:23:01.001180 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:01.001182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:01.001863 (+ 681us) server_negotiation.cc:407] Connection header received
0928 06:23:01.001989 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:01.001990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:01.002000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:01.002030 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:01.002241 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.002963 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.004397 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.005069 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.005116 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:01.005125 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:01.005431 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 06:23:01.005483 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:23:01.005743 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9858605c89ae4f5c9e8c36030f4de57c" candidate_uuid: "890c8688de7f4a2f998c2520df9a3a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5349192348940f99ca4e6f57033b584" is_pre_election: true
I20260928 06:23:01.005898 2014 raft_consensus.cc:2454] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890c8688de7f4a2f998c2520df9a3a17 in term 0.
I20260928 06:23:01.006139 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:01.001014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:35817 (local address 127.31.214.66:36383)
0928 06:23:01.001791 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:01.001803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:01.001891 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:01.002095 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:01.002095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:01.002097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:01.002170 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:01.002172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.003077 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.003078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:01.004286 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:01.004289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.005268 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.005269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:01.005354 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:01.005358 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:23:01.006012 (+ 654us) client_negotiation.cc:240] Negotiation successful
0928 06:23:01.006058 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":670,"threads_started":1}
I20260928 06:23:01.006141 2085 leader_election.cc:302] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890c8688de7f4a2f998c2520df9a3a17, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:01.006522 2368 raft_consensus.cc:2781] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:01.006577 2368 raft_consensus.cc:482] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:01.006603 2368 raft_consensus.cc:3037] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:01.007688 2368 raft_consensus.cc:504] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:01.007861 2368 leader_election.cc:288] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [CANDIDATE]: Term 1 election: Requested vote from peers b5349192348940f99ca4e6f57033b584 (127.31.214.65:43391), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:01.007956 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9858605c89ae4f5c9e8c36030f4de57c" candidate_uuid: "890c8688de7f4a2f998c2520df9a3a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6" is_pre_election: true
I20260928 06:23:01.008045 2294 raft_consensus.cc:2454] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890c8688de7f4a2f998c2520df9a3a17 in term 0.
I20260928 06:23:01.008493 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9858605c89ae4f5c9e8c36030f4de57c" candidate_uuid: "890c8688de7f4a2f998c2520df9a3a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6"
I20260928 06:23:01.008507 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9858605c89ae4f5c9e8c36030f4de57c" candidate_uuid: "890c8688de7f4a2f998c2520df9a3a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5349192348940f99ca4e6f57033b584"
I20260928 06:23:01.008579 2294 raft_consensus.cc:3037] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:01.008602 2014 raft_consensus.cc:3037] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:01.009361 2014 raft_consensus.cc:2454] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890c8688de7f4a2f998c2520df9a3a17 in term 1.
I20260928 06:23:01.009362 2294 raft_consensus.cc:2454] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890c8688de7f4a2f998c2520df9a3a17 in term 1.
I20260928 06:23:01.009586 2085 leader_election.cc:302] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890c8688de7f4a2f998c2520df9a3a17, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:01.009719 2368 raft_consensus.cc:2781] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:01.009884 2368 raft_consensus.cc:686] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 1 LEADER]: Becoming Leader. State: Replica: 890c8688de7f4a2f998c2520df9a3a17, State: Running, Role: LEADER
I20260928 06:23:01.010004 2368 consensus_queue.cc:237] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:01.010885 1857 catalog_manager.cc:5495] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66). New cstate: current_term: 1 leader_uuid: "890c8688de7f4a2f998c2520df9a3a17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:01.019140 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:01.016150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.31.214.66:34531)
0928 06:23:01.016317 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:01.016318 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:01.016325 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:23:01.016369 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:01.016370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:01.016388 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:01.016481 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:01.016587 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.017076 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.017828 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.018162 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.018226 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:01.018743 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:01.018870 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:01.018954 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:01.019008 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:23:01.019030 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
I20260928 06:23:01.025121 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:01.022173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:57493 (local address 127.31.214.65:43391)
0928 06:23:01.022253 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:01.022255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:01.022349 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:23:01.022378 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:01.022379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:01.022393 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:01.022453 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:01.022687 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.023443 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.024217 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.024527 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.024605 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:01.024807 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:01.024845 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:01.024899 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:01.025025 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:23:01.025059 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:23:01.025173 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:01.022107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:43391 (local address 127.31.214.66:57493)
0928 06:23:01.022263 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:01.022281 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:01.022337 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:01.022469 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:01.022470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:01.022471 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:23:01.022610 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:01.022612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.023522 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.023522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:01.024125 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:01.024127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.024649 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.024650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:01.024699 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:01.024715 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:23:01.024937 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:23:01.024940 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:23:01.025012 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:23:01.025064 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260928 06:23:01.027652 2014 raft_consensus.cc:1261] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Refusing update from remote peer 890c8688de7f4a2f998c2520df9a3a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:01.027868 2294 raft_consensus.cc:1261] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Refusing update from remote peer 890c8688de7f4a2f998c2520df9a3a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:01.027973 2368 consensus_queue.cc:1035] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:01.028110 2415 consensus_queue.cc:1035] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:01.037197 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:01.034312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:46007 (local address 127.31.214.66:34531)
0928 06:23:01.034358 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:01.034361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:01.034369 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:23:01.034393 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:01.034393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:01.034405 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:01.034459 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:01.034631 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.035288 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.036446 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.036724 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.036792 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:01.036972 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:01.037002 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:01.037045 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:01.037121 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:23:01.037141 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:23:01.037250 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:01.034216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:34531 (local address 127.31.214.65:46007)
0928 06:23:01.034274 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:01.034288 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:01.034331 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:01.034514 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:01.034515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:01.034516 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:23:01.034565 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:01.034567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.035390 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.035391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:01.036338 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:01.036341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:01.036829 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:01.036830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:01.036884 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:01.036891 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:23:01.037071 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:23:01.037073 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:23:01.037121 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:23:01.037170 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:23:01.037581 2391 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:23:02.559898 1857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:02.560024 1857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:02.561093 1857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9: Sending DeleteTablet for 3 replicas of tablet 9858605c89ae4f5c9e8c36030f4de57c
I20260928 06:23:02.561569 2134 tablet_service.cc:1514] Processing DeleteTablet for tablet 9858605c89ae4f5c9e8c36030f4de57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:02 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:36324
I20260928 06:23:02.561595 1993 tablet_service.cc:1514] Processing DeleteTablet for tablet 9858605c89ae4f5c9e8c36030f4de57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:02 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:52010
I20260928 06:23:02.561636 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 9858605c89ae4f5c9e8c36030f4de57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:02 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:41704
I20260928 06:23:02.561849 2448 tablet_replica.cc:330] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: stopping tablet replica
I20260928 06:23:02.561931 2448 raft_consensus.cc:2229] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:02.561998 2448 raft_consensus.cc:2258] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:02.562112 2447 tablet_replica.cc:330] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: stopping tablet replica
I20260928 06:23:02.562181 2447 raft_consensus.cc:2229] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:23:02.562276 2447 raft_consensus.cc:2258] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:02.562638 2447 ts_tablet_manager.cc:1905] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:02.564791 2449 tablet_replica.cc:330] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: stopping tablet replica
I20260928 06:23:02.564813 2448 ts_tablet_manager.cc:1905] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:02.564880 2449 raft_consensus.cc:2229] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:02.564924 2449 raft_consensus.cc:2258] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:02.564950 2447 ts_tablet_manager.cc:1918] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:23:02.565042 2447 log.cc:1196] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/9858605c89ae4f5c9e8c36030f4de57c
I20260928 06:23:02.565196 2449 ts_tablet_manager.cc:1905] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:02.565363 2447 ts_tablet_manager.cc:1939] T 9858605c89ae4f5c9e8c36030f4de57c P 890c8688de7f4a2f998c2520df9a3a17: Deleting consensus metadata
I20260928 06:23:02.565826 1841 catalog_manager.cc:4841] TS 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531): tablet 9858605c89ae4f5c9e8c36030f4de57c (table test-table [id=6db591c75711454ab200a96fb2e501a5]) successfully deleted
I20260928 06:23:02.566636 2448 ts_tablet_manager.cc:1918] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:23:02.566720 2448 log.cc:1196] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/9858605c89ae4f5c9e8c36030f4de57c
I20260928 06:23:02.566886 2449 ts_tablet_manager.cc:1918] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:23:02.566936 2448 ts_tablet_manager.cc:1939] T 9858605c89ae4f5c9e8c36030f4de57c P b5349192348940f99ca4e6f57033b584: Deleting consensus metadata
I20260928 06:23:02.566951 2449 log.cc:1196] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/9858605c89ae4f5c9e8c36030f4de57c
I20260928 06:23:02.567176 2449 ts_tablet_manager.cc:1939] T 9858605c89ae4f5c9e8c36030f4de57c P 672ee895b5454350b7cfee7aa77f3cc6: Deleting consensus metadata
I20260928 06:23:02.567497 1841 catalog_manager.cc:4841] TS b5349192348940f99ca4e6f57033b584 (127.31.214.65:43391): tablet 9858605c89ae4f5c9e8c36030f4de57c (table test-table [id=6db591c75711454ab200a96fb2e501a5]) successfully deleted
I20260928 06:23:02.567605 1841 catalog_manager.cc:4841] TS 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817): tablet 9858605c89ae4f5c9e8c36030f4de57c (table test-table [id=6db591c75711454ab200a96fb2e501a5]) successfully deleted
Sep 28 06:23:02 dist-test-slave-q03f krb5kdc[1812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576582, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:23:02 dist-test-slave-q03f krb5kdc[1812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576582, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:02.598850 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:02.581302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.31.214.126:33903)
0928 06:23:02.581497 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:02.581499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:02.581515 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:23:02.581638 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:02.581639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:02.581652 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:02.581742 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:02.582520 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:02.583136 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:02.583945 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:02.584175 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:02.584277 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:02.596035 (+ 11758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:02.596038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:02.596039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:02.596050 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:02.597556 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:02.597963 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:02.597965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:02.597965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:02.598038 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:02.598253 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:02.598255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:02.598255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:02.598400 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:02.598522 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:02.598634 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 06:23:02.598732 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260928 06:23:02.600658 1857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:02.600859 1857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:02.602922 2274 tablet_service.cc:1467] Processing CreateTablet for tablet eba2de67acad4db8a63a2c7ea964ae0c (DEFAULT_TABLE table=test-table [id=12ef940617cf463ebb30bf693e9b45c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:02.602887 2134 tablet_service.cc:1467] Processing CreateTablet for tablet eba2de67acad4db8a63a2c7ea964ae0c (DEFAULT_TABLE table=test-table [id=12ef940617cf463ebb30bf693e9b45c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:02.602949 1993 tablet_service.cc:1467] Processing CreateTablet for tablet eba2de67acad4db8a63a2c7ea964ae0c (DEFAULT_TABLE table=test-table [id=12ef940617cf463ebb30bf693e9b45c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:02.603078 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba2de67acad4db8a63a2c7ea964ae0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:02.603078 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba2de67acad4db8a63a2c7ea964ae0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:02.603111 1993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba2de67acad4db8a63a2c7ea964ae0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:02.604507 2458 tablet_bootstrap.cc:492] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: Bootstrap starting.
I20260928 06:23:02.604681 2459 tablet_bootstrap.cc:492] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: Bootstrap starting.
I20260928 06:23:02.604795 2460 tablet_bootstrap.cc:492] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: Bootstrap starting.
I20260928 06:23:02.605069 2458 tablet_bootstrap.cc:654] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:02.605077 2459 tablet_bootstrap.cc:654] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:02.605233 2460 tablet_bootstrap.cc:654] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:02.605700 2458 tablet_bootstrap.cc:492] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: No bootstrap required, opened a new log
I20260928 06:23:02.605780 2458 ts_tablet_manager.cc:1397] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:02.606130 2460 tablet_bootstrap.cc:492] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: No bootstrap required, opened a new log
I20260928 06:23:02.606197 2460 ts_tablet_manager.cc:1397] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:02.606181 2458 raft_consensus.cc:348] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:02.606284 2458 raft_consensus.cc:374] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:02.606309 2458 raft_consensus.cc:729] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5349192348940f99ca4e6f57033b584, State: Initialized, Role: FOLLOWER
I20260928 06:23:02.606353 2458 consensus_queue.cc:260] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:02.606326 2460 raft_consensus.cc:348] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:02.606403 2460 raft_consensus.cc:374] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:02.606442 2460 raft_consensus.cc:729] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890c8688de7f4a2f998c2520df9a3a17, State: Initialized, Role: FOLLOWER
I20260928 06:23:02.606455 2458 ts_tablet_manager.cc:1428] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:02.606496 2460 consensus_queue.cc:260] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:02.606645 2460 ts_tablet_manager.cc:1428] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:02.606752 2459 tablet_bootstrap.cc:492] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: No bootstrap required, opened a new log
I20260928 06:23:02.606971 2459 ts_tablet_manager.cc:1397] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:02.607122 2459 raft_consensus.cc:348] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:02.607211 2459 raft_consensus.cc:374] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:02.607239 2459 raft_consensus.cc:729] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672ee895b5454350b7cfee7aa77f3cc6, State: Initialized, Role: FOLLOWER
I20260928 06:23:02.607285 2459 consensus_queue.cc:260] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:02.607475 2459 ts_tablet_manager.cc:1428] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:02.954669 2428 raft_consensus.cc:482] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:02.954816 2428 raft_consensus.cc:504] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:02.955015 2428 leader_election.cc:288] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:02.955336 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba2de67acad4db8a63a2c7ea964ae0c" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6" is_pre_election: true
I20260928 06:23:02.955320 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba2de67acad4db8a63a2c7ea964ae0c" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890c8688de7f4a2f998c2520df9a3a17" is_pre_election: true
I20260928 06:23:02.955469 2294 raft_consensus.cc:2454] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 0.
I20260928 06:23:02.955466 2154 raft_consensus.cc:2454] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 0.
I20260928 06:23:02.955675 1945 leader_election.cc:302] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890c8688de7f4a2f998c2520df9a3a17, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:02.955778 2428 raft_consensus.cc:2781] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:02.955821 2428 raft_consensus.cc:482] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:02.955849 2428 raft_consensus.cc:3037] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:02.956600 2428 raft_consensus.cc:504] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:02.956738 2428 leader_election.cc:288] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 election: Requested vote from peers 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:02.956899 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba2de67acad4db8a63a2c7ea964ae0c" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890c8688de7f4a2f998c2520df9a3a17"
I20260928 06:23:02.956933 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba2de67acad4db8a63a2c7ea964ae0c" candidate_uuid: "b5349192348940f99ca4e6f57033b584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6"
I20260928 06:23:02.957008 2154 raft_consensus.cc:3037] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:02.957006 2294 raft_consensus.cc:3037] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:02.957710 2294 raft_consensus.cc:2454] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 1.
I20260928 06:23:02.957883 2154 raft_consensus.cc:2454] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5349192348940f99ca4e6f57033b584 in term 1.
I20260928 06:23:02.957903 1945 leader_election.cc:302] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672ee895b5454350b7cfee7aa77f3cc6, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:02.958036 2428 raft_consensus.cc:2781] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:02.958091 2428 raft_consensus.cc:686] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 1 LEADER]: Becoming Leader. State: Replica: b5349192348940f99ca4e6f57033b584, State: Running, Role: LEADER
I20260928 06:23:02.958149 2428 consensus_queue.cc:237] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:02.958760 1857 catalog_manager.cc:5495] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5349192348940f99ca4e6f57033b584 (127.31.214.65). New cstate: current_term: 1 leader_uuid: "b5349192348940f99ca4e6f57033b584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:03.052215 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:03.049824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.31.214.65:43391)
0928 06:23:03.050021 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:03.050023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:03.050067 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:23:03.050088 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:03.050089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:03.050099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:03.050158 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:03.050278 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:03.050780 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:03.051585 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:03.051780 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:03.051837 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:03.051958 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:03.051987 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:03.052026 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:03.052075 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 06:23:03.052093 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":143,"threads_started":1}
I20260928 06:23:03.055014 2294 raft_consensus.cc:1261] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Refusing update from remote peer b5349192348940f99ca4e6f57033b584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:03.055101 2154 raft_consensus.cc:1261] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Refusing update from remote peer b5349192348940f99ca4e6f57033b584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:03.055222 2428 consensus_queue.cc:1035] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:03.055323 2433 consensus_queue.cc:1035] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:03.065871 1857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:03.066035 1857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:03.066782 1857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9: Sending DeleteTablet for 3 replicas of tablet eba2de67acad4db8a63a2c7ea964ae0c
I20260928 06:23:03.067029 1993 tablet_service.cc:1514] Processing DeleteTablet for tablet eba2de67acad4db8a63a2c7ea964ae0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:03 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:52010
I20260928 06:23:03.067090 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet eba2de67acad4db8a63a2c7ea964ae0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:03 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:41704
I20260928 06:23:03.067157 2448 tablet_replica.cc:330] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: stopping tablet replica
I20260928 06:23:03.067162 2134 tablet_service.cc:1514] Processing DeleteTablet for tablet eba2de67acad4db8a63a2c7ea964ae0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:03 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:36324
I20260928 06:23:03.067229 2448 raft_consensus.cc:2229] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:23:03.067298 2448 raft_consensus.cc:2258] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:03.067339 2472 tablet_replica.cc:330] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: stopping tablet replica
I20260928 06:23:03.067385 2449 tablet_replica.cc:330] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: stopping tablet replica
I20260928 06:23:03.067396 2472 raft_consensus.cc:2229] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:03.067438 2472 raft_consensus.cc:2258] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:03.067448 2449 raft_consensus.cc:2229] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:03.067490 2449 raft_consensus.cc:2258] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:03.067576 1857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:03.067737 1857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:03.067801 2448 ts_tablet_manager.cc:1905] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:03.067852 2449 ts_tablet_manager.cc:1905] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:03.067881 2472 ts_tablet_manager.cc:1905] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:03.069571 1993 tablet_service.cc:1467] Processing CreateTablet for tablet b3cfb00df8af44559b63484537a9cdcc (DEFAULT_TABLE table=test-table [id=2ac7b15fa7af42bd9ad7807a9ea0ae4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:03.069597 2134 tablet_service.cc:1467] Processing CreateTablet for tablet b3cfb00df8af44559b63484537a9cdcc (DEFAULT_TABLE table=test-table [id=2ac7b15fa7af42bd9ad7807a9ea0ae4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:03.069628 2274 tablet_service.cc:1467] Processing CreateTablet for tablet b3cfb00df8af44559b63484537a9cdcc (DEFAULT_TABLE table=test-table [id=2ac7b15fa7af42bd9ad7807a9ea0ae4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:03.069769 2472 ts_tablet_manager.cc:1918] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:03.069769 2448 ts_tablet_manager.cc:1918] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:03.069792 2449 ts_tablet_manager.cc:1918] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:03.069820 2448 log.cc:1196] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/eba2de67acad4db8a63a2c7ea964ae0c
I20260928 06:23:03.069820 2472 log.cc:1196] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/eba2de67acad4db8a63a2c7ea964ae0c
I20260928 06:23:03.069829 2449 log.cc:1196] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/eba2de67acad4db8a63a2c7ea964ae0c
I20260928 06:23:03.070081 2472 ts_tablet_manager.cc:1939] T eba2de67acad4db8a63a2c7ea964ae0c P 890c8688de7f4a2f998c2520df9a3a17: Deleting consensus metadata
I20260928 06:23:03.070081 2448 ts_tablet_manager.cc:1939] T eba2de67acad4db8a63a2c7ea964ae0c P b5349192348940f99ca4e6f57033b584: Deleting consensus metadata
I20260928 06:23:03.070088 2449 ts_tablet_manager.cc:1939] T eba2de67acad4db8a63a2c7ea964ae0c P 672ee895b5454350b7cfee7aa77f3cc6: Deleting consensus metadata
I20260928 06:23:03.070240 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3cfb00df8af44559b63484537a9cdcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:03.070257 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3cfb00df8af44559b63484537a9cdcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:03.070506 1841 catalog_manager.cc:4841] TS 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817): tablet eba2de67acad4db8a63a2c7ea964ae0c (table test-table [id=12ef940617cf463ebb30bf693e9b45c2]) successfully deleted
I20260928 06:23:03.071256 2460 tablet_bootstrap.cc:492] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: Bootstrap starting.
I20260928 06:23:03.071290 1993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3cfb00df8af44559b63484537a9cdcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:03.071385 2459 tablet_bootstrap.cc:492] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: Bootstrap starting.
I20260928 06:23:03.071602 2460 tablet_bootstrap.cc:654] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:03.071961 2459 tablet_bootstrap.cc:654] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:03.072178 2460 tablet_bootstrap.cc:492] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: No bootstrap required, opened a new log
I20260928 06:23:03.072233 2460 ts_tablet_manager.cc:1397] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:03.072396 1841 catalog_manager.cc:4841] TS 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531): tablet eba2de67acad4db8a63a2c7ea964ae0c (table test-table [id=12ef940617cf463ebb30bf693e9b45c2]) successfully deleted
I20260928 06:23:03.072365 2460 raft_consensus.cc:348] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:03.072458 2460 raft_consensus.cc:374] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:03.072494 2460 raft_consensus.cc:729] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890c8688de7f4a2f998c2520df9a3a17, State: Initialized, Role: FOLLOWER
I20260928 06:23:03.072542 2460 consensus_queue.cc:260] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:03.072597 2458 tablet_bootstrap.cc:492] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584: Bootstrap starting.
I20260928 06:23:03.072659 2460 ts_tablet_manager.cc:1428] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:03.072911 2458 tablet_bootstrap.cc:654] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:03.073175 2459 tablet_bootstrap.cc:492] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: No bootstrap required, opened a new log
I20260928 06:23:03.073236 2459 ts_tablet_manager.cc:1397] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:03.073268 1841 catalog_manager.cc:4841] TS b5349192348940f99ca4e6f57033b584 (127.31.214.65:43391): tablet eba2de67acad4db8a63a2c7ea964ae0c (table test-table [id=12ef940617cf463ebb30bf693e9b45c2]) successfully deleted
I20260928 06:23:03.073393 2459 raft_consensus.cc:348] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:03.073509 2459 raft_consensus.cc:374] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:03.073540 2459 raft_consensus.cc:729] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672ee895b5454350b7cfee7aa77f3cc6, State: Initialized, Role: FOLLOWER
I20260928 06:23:03.073586 2459 consensus_queue.cc:260] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:03.073612 2458 tablet_bootstrap.cc:492] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584: No bootstrap required, opened a new log
I20260928 06:23:03.073657 2458 ts_tablet_manager.cc:1397] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:03.073693 2459 ts_tablet_manager.cc:1428] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:03.073766 2458 raft_consensus.cc:348] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:03.073858 2458 raft_consensus.cc:374] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:03.073882 2458 raft_consensus.cc:729] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5349192348940f99ca4e6f57033b584, State: Initialized, Role: FOLLOWER
I20260928 06:23:03.073935 2458 consensus_queue.cc:260] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:03.074101 2458 ts_tablet_manager.cc:1428] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:03.303710 2441 raft_consensus.cc:482] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:03.303822 2441 raft_consensus.cc:504] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:03.304049 2441 leader_election.cc:288] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5349192348940f99ca4e6f57033b584 (127.31.214.65:43391), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:03.304361 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cfb00df8af44559b63484537a9cdcc" candidate_uuid: "890c8688de7f4a2f998c2520df9a3a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6" is_pre_election: true
I20260928 06:23:03.304329 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cfb00df8af44559b63484537a9cdcc" candidate_uuid: "890c8688de7f4a2f998c2520df9a3a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5349192348940f99ca4e6f57033b584" is_pre_election: true
I20260928 06:23:03.304457 2293 raft_consensus.cc:2454] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890c8688de7f4a2f998c2520df9a3a17 in term 0.
I20260928 06:23:03.304468 2014 raft_consensus.cc:2454] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890c8688de7f4a2f998c2520df9a3a17 in term 0.
I20260928 06:23:03.304646 2085 leader_election.cc:302] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890c8688de7f4a2f998c2520df9a3a17, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:03.304783 2441 raft_consensus.cc:2781] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:03.304837 2441 raft_consensus.cc:482] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:03.304865 2441 raft_consensus.cc:3037] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:03.305686 2441 raft_consensus.cc:504] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:03.305835 2441 leader_election.cc:288] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [CANDIDATE]: Term 1 election: Requested vote from peers b5349192348940f99ca4e6f57033b584 (127.31.214.65:43391), 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817)
I20260928 06:23:03.306067 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cfb00df8af44559b63484537a9cdcc" candidate_uuid: "890c8688de7f4a2f998c2520df9a3a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5349192348940f99ca4e6f57033b584"
I20260928 06:23:03.306154 2014 raft_consensus.cc:3037] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:03.306100 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cfb00df8af44559b63484537a9cdcc" candidate_uuid: "890c8688de7f4a2f998c2520df9a3a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672ee895b5454350b7cfee7aa77f3cc6"
I20260928 06:23:03.306229 2293 raft_consensus.cc:3037] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:03.306847 2014 raft_consensus.cc:2454] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890c8688de7f4a2f998c2520df9a3a17 in term 1.
I20260928 06:23:03.306979 2293 raft_consensus.cc:2454] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890c8688de7f4a2f998c2520df9a3a17 in term 1.
I20260928 06:23:03.307027 2085 leader_election.cc:302] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890c8688de7f4a2f998c2520df9a3a17, b5349192348940f99ca4e6f57033b584; no voters:
I20260928 06:23:03.307111 2441 raft_consensus.cc:2781] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:03.307245 2441 raft_consensus.cc:686] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 1 LEADER]: Becoming Leader. State: Replica: 890c8688de7f4a2f998c2520df9a3a17, State: Running, Role: LEADER
I20260928 06:23:03.307314 2441 consensus_queue.cc:237] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } }
I20260928 06:23:03.308059 1856 catalog_manager.cc:5495] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66). New cstate: current_term: 1 leader_uuid: "890c8688de7f4a2f998c2520df9a3a17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "890c8688de7f4a2f998c2520df9a3a17" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:03.359913 2478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:03.357321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.31.214.66:34531)
0928 06:23:03.357503 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:03.357505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:03.357511 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:23:03.357530 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:03.357531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:03.357540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:03.357614 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:03.357727 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:03.358271 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:03.359130 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:03.359377 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:03.359445 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:03.359631 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:03.359672 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:03.359733 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:03.359787 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:23:03.359809 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260928 06:23:03.362658 2014 raft_consensus.cc:1261] T b3cfb00df8af44559b63484537a9cdcc P b5349192348940f99ca4e6f57033b584 [term 1 FOLLOWER]: Refusing update from remote peer 890c8688de7f4a2f998c2520df9a3a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:03.362658 2293 raft_consensus.cc:1261] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Refusing update from remote peer 890c8688de7f4a2f998c2520df9a3a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:03.362926 2476 consensus_queue.cc:1035] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5349192348940f99ca4e6f57033b584" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:03.363041 2441 consensus_queue.cc:1035] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672ee895b5454350b7cfee7aa77f3cc6" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:23:03.367717 2430 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:23:04.887531 1857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:04.887653 1857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:04.888653 1857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 414053a253b4476d8f4ce95ec44e8cb9: Sending DeleteTablet for 3 replicas of tablet b3cfb00df8af44559b63484537a9cdcc
I20260928 06:23:04.888897 1993 tablet_service.cc:1514] Processing DeleteTablet for tablet b3cfb00df8af44559b63484537a9cdcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:04 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:52010
I20260928 06:23:04.888950 2134 tablet_service.cc:1514] Processing DeleteTablet for tablet b3cfb00df8af44559b63484537a9cdcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:04 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:36324
I20260928 06:23:04.888993 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet b3cfb00df8af44559b63484537a9cdcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:04 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:41704
I20260928 06:23:04.889055 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 1929
I20260928 06:23:04.889784 2507 tablet_replica.cc:330] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: stopping tablet replica
I20260928 06:23:04.889850 2507 raft_consensus.cc:2229] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:04.889892 2507 raft_consensus.cc:2258] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:04.890278 2507 ts_tablet_manager.cc:1905] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:04.892344 2507 ts_tablet_manager.cc:1918] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:23:04.892414 2507 log.cc:1196] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/b3cfb00df8af44559b63484537a9cdcc
I20260928 06:23:04.892661 2507 ts_tablet_manager.cc:1939] T b3cfb00df8af44559b63484537a9cdcc P 672ee895b5454350b7cfee7aa77f3cc6: Deleting consensus metadata
I20260928 06:23:04.893322 2506 tablet_replica.cc:330] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: stopping tablet replica
I20260928 06:23:04.893352 1841 catalog_manager.cc:4841] TS 672ee895b5454350b7cfee7aa77f3cc6 (127.31.214.67:35817): tablet b3cfb00df8af44559b63484537a9cdcc (table test-table [id=2ac7b15fa7af42bd9ad7807a9ea0ae4e]) successfully deleted
I20260928 06:23:04.893404 2506 raft_consensus.cc:2229] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:23:04.893481 2506 raft_consensus.cc:2258] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:04.893741 2506 ts_tablet_manager.cc:1905] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:04.895716 2506 ts_tablet_manager.cc:1918] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:23:04.895784 2506 log.cc:1196] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/b3cfb00df8af44559b63484537a9cdcc
I20260928 06:23:04.896039 2506 ts_tablet_manager.cc:1939] T b3cfb00df8af44559b63484537a9cdcc P 890c8688de7f4a2f998c2520df9a3a17: Deleting consensus metadata
I20260928 06:23:04.896672 1841 catalog_manager.cc:4841] TS 890c8688de7f4a2f998c2520df9a3a17 (127.31.214.66:34531): tablet b3cfb00df8af44559b63484537a9cdcc (table test-table [id=2ac7b15fa7af42bd9ad7807a9ea0ae4e]) successfully deleted
W20260928 06:23:04.899312 1841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:43391 (error 108)
I20260928 06:23:04.899555 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 2069
I20260928 06:23:04.900791 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:04.900542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:43391 (local address 127.0.0.1:52044)
0928 06:23:04.900696 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:04.900753 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:43391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
W20260928 06:23:04.900904 1841 catalog_manager.cc:4584] TS b5349192348940f99ca4e6f57033b584 (127.31.214.65:43391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3cfb00df8af44559b63484537a9cdcc: Network error: Client connection negotiation failed: client connection to 127.31.214.65:43391: connect: Connection refused (error 111)
I20260928 06:23:04.905769 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 2209
I20260928 06:23:04.910774 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 1825
2026-09-28T06:23:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6548 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:04 dist-test-slave-q03f krb5kdc[2512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:04 dist-test-slave-q03f krb5kdc[2512](info): set up 2 sockets
Sep 28 06:23:04 dist-test-slave-q03f krb5kdc[2512](info): commencing operation
krb5kdc: starting...
W20260928 06:23:05.994330 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[2512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:23:06.126070 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:32807
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:32807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:06.209537 2525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:06.209802 2525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:06.209846 2525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:06.211043 2525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:06.211117 2525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:06.211143 2525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:06.211170 2525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:06.211201 2525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:06.212699 2525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:32807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:06.212980 2525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:06.213239 2525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:23:06.218914 2525 server_base.cc:1034] running on GCE node
W20260928 06:23:06.218840 2533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:06.218881 2530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:06.218871 2531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:06.219398 2525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:06.219712 2525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:06.220880 2525 hybrid_clock.cc:648] HybridClock initialized: now 1790576586220876 us; error 39 us; skew 500 ppm
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[2512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:06.223516 2525 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:23:06.223987 2525 webserver.cc:460] Webserver started at http://127.31.214.126:36303/ using document root <none> and password file <none>
I20260928 06:23:06.224222 2525 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:06.224291 2525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:06.224447 2525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:06.225577 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "7a13523134034a10ab7a8ffd7f855f3d"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "46aecb731f344b9140ec12dd018dac84"
server_key_iv: "4452b95e0fb6264320ccb966ae0a80a3"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.226012 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "7a13523134034a10ab7a8ffd7f855f3d"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "46aecb731f344b9140ec12dd018dac84"
server_key_iv: "4452b95e0fb6264320ccb966ae0a80a3"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.227473 2525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:06.228341 2540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.228571 2525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:06.228679 2525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "7a13523134034a10ab7a8ffd7f855f3d"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "46aecb731f344b9140ec12dd018dac84"
server_key_iv: "4452b95e0fb6264320ccb966ae0a80a3"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.228770 2525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:06.245065 2525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:06.247834 2525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:06.248062 2525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:06.252529 2525 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:32807
I20260928 06:23:06.252702 2592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:32807 every 8 connection(s)
I20260928 06:23:06.253006 2525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:06.253933 2593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:06.256464 2593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d: Bootstrap starting.
I20260928 06:23:06.257125 2593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:06.257450 2593 log.cc:824] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:06.258285 2593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d: No bootstrap required, opened a new log
I20260928 06:23:06.259944 2593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a13523134034a10ab7a8ffd7f855f3d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 32807 } }
I20260928 06:23:06.260099 2593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:06.260123 2593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a13523134034a10ab7a8ffd7f855f3d, State: Initialized, Role: FOLLOWER
I20260928 06:23:06.260275 2593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a13523134034a10ab7a8ffd7f855f3d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 32807 } }
I20260928 06:23:06.260357 2593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:06.260413 2593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:06.260468 2593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:06.261106 2593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a13523134034a10ab7a8ffd7f855f3d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 32807 } }
I20260928 06:23:06.261222 2593 leader_election.cc:302] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a13523134034a10ab7a8ffd7f855f3d; no voters:
I20260928 06:23:06.261443 2593 leader_election.cc:288] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:06.261475 2596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:06.261538 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 2525
I20260928 06:23:06.261605 2596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [term 1 LEADER]: Becoming Leader. State: Replica: 7a13523134034a10ab7a8ffd7f855f3d, State: Running, Role: LEADER
I20260928 06:23:06.261650 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:06.261682 2596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a13523134034a10ab7a8ffd7f855f3d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 32807 } }
I20260928 06:23:06.261739 2593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:06.261822 32601 external_mini_cluster.cc:1428] Setting key 6c84e159351e61bb6ac638f72ba786ae
I20260928 06:23:06.262230 2598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a13523134034a10ab7a8ffd7f855f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a13523134034a10ab7a8ffd7f855f3d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 32807 } } }
I20260928 06:23:06.262327 2598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:06.262936 2597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a13523134034a10ab7a8ffd7f855f3d. Latest consensus state: current_term: 1 leader_uuid: "7a13523134034a10ab7a8ffd7f855f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a13523134034a10ab7a8ffd7f855f3d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 32807 } } }
I20260928 06:23:06.263043 2597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d [sys.catalog]: This master's current role is: LEADER
W20260928 06:23:06.263700 2614 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:23:06.263772 2614 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:23:06.263830 2606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:06.264048 2606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:06.265620 2606 catalog_manager.cc:1347] Generated new cluster ID: a80529d7dc3142e092f9ee1808cd3401
I20260928 06:23:06.265676 2606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[2512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:06.272974 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:06.262527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.31.214.126:32807)
0928 06:23:06.262766 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:06.262777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:06.262827 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:23:06.262971 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:06.262979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:06.263046 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:06.263221 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:06.264026 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.265394 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.266359 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.266797 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.266918 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:06.269359 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:06.269363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:06.269364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:06.269377 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:06.271330 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:06.271703 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:06.271706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:06.271707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:06.271824 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:06.272047 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:06.272048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:06.272048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:06.272326 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:06.272507 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:06.272590 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:23:06.272735 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260928 06:23:06.278803 2606 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:06.279402 2606 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:06.283867 2606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a13523134034a10ab7a8ffd7f855f3d: Generated new TSK 0
I20260928 06:23:06.284065 2606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:23:06.319294 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:32807
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:06.410147 2619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:06.410379 2619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:06.410408 2619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:06.411645 2619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:06.411682 2619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:06.411710 2619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:06.413158 2619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:06.413390 2619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:06.413591 2619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:06.419421 2624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:06.419628 2619 server_base.cc:1034] running on GCE node
W20260928 06:23:06.419414 2627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:06.419415 2625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:06.419986 2619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:06.420251 2619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:06.421429 2619 hybrid_clock.cc:648] HybridClock initialized: now 1790576586421402 us; error 46 us; skew 500 ppm
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[2512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:06.424249 2619 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:23:06.424782 2619 webserver.cc:460] Webserver started at http://127.31.214.65:45443/ using document root <none> and password file <none>
I20260928 06:23:06.424983 2619 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:06.425024 2619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:06.425117 2619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:06.426230 2619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "cffb193ceda04e3c8397087c9ae9f3c8"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "5968114466b4be28118364a1537379b0"
server_key_iv: "7b37820922b661b8239a2f69875c86ad"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.426577 2619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "cffb193ceda04e3c8397087c9ae9f3c8"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "5968114466b4be28118364a1537379b0"
server_key_iv: "7b37820922b661b8239a2f69875c86ad"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.427939 2619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:23:06.428753 2634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.429013 2619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:06.429113 2619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "cffb193ceda04e3c8397087c9ae9f3c8"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "5968114466b4be28118364a1537379b0"
server_key_iv: "7b37820922b661b8239a2f69875c86ad"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.429174 2619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:06.449141 2619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:06.451836 2619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:06.451977 2619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:06.452200 2619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:06.452561 2619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:06.452600 2619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.452646 2619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:06.452682 2619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.459086 2619 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:34405
I20260928 06:23:06.459173 2747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:34405 every 8 connection(s)
I20260928 06:23:06.459522 2619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:23:06.465219 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 2619
I20260928 06:23:06.465342 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:23:06.465543 32601 external_mini_cluster.cc:1428] Setting key 73423b6e4c9e94023ba94e8b7959539a
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[2512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:06.470419 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:06.460334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:55569 (local address 127.31.214.126:32807)
0928 06:23:06.460408 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:06.460410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:06.460672 (+ 262us) server_negotiation.cc:407] Connection header received
0928 06:23:06.461160 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:06.461161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:06.461175 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:06.461243 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:06.462401 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.463031 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.464076 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.464282 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.464344 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:06.467212 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:06.467214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:06.467214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:06.467222 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:06.468829 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:06.469312 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:06.469313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:06.469314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:06.469380 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:06.469665 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:06.469666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:06.469666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:06.469793 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:06.469879 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:06.470229 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 06:23:06.470322 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:23:06.470649 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:06.460303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:32807 (local address 127.31.214.65:55569)
0928 06:23:06.460513 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:06.460561 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:06.461049 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:06.461323 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:06.461331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:06.462082 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:06.462289 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:06.462295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.463146 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.463146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:06.463979 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:06.463982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.464365 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.464365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:06.464509 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:06.465099 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:06.465111 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:06.467089 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:06.468956 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:06.468965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:06.468967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:06.469214 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:06.469473 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:06.469478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:06.469478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:06.469587 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:06.469860 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:06.469868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:06.470081 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 06:23:06.470195 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:23:06.470389 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260928 06:23:06.471386 2748 heartbeater.cc:347] Connected to a master server at 127.31.214.126:32807
I20260928 06:23:06.471508 2748 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:06.471706 2748 heartbeater.cc:511] Master 127.31.214.126:32807 requested a full tablet report, sending...
I20260928 06:23:06.472359 2557 ts_manager.cc:194] Registered new tserver with Master: cffb193ceda04e3c8397087c9ae9f3c8 (127.31.214.65:34405)
I20260928 06:23:06.472996 2557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:55569
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:23:06.493590 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:32807
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:06.577702 2753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:06.577919 2753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:06.577965 2753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:06.579195 2753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:06.579239 2753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:06.579267 2753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:06.580641 2753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:06.580835 2753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:06.581007 2753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:06.587065 2758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:06.587133 2761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:06.587065 2759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:06.587409 2753 server_base.cc:1034] running on GCE node
I20260928 06:23:06.587584 2753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:06.587809 2753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:06.588955 2753 hybrid_clock.cc:648] HybridClock initialized: now 1790576586588935 us; error 36 us; skew 500 ppm
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[2512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:06.591670 2753 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:23:06.592190 2753 webserver.cc:460] Webserver started at http://127.31.214.66:36175/ using document root <none> and password file <none>
I20260928 06:23:06.592473 2753 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:06.592584 2753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:06.592742 2753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:06.594081 2753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "ef9c4e9c4eaa453a882d055a50b98a72"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "210a68820808a501b3551bfa239aba25"
server_key_iv: "6422a6b20a9519f4d3555d20651509d8"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.594465 2753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "ef9c4e9c4eaa453a882d055a50b98a72"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "210a68820808a501b3551bfa239aba25"
server_key_iv: "6422a6b20a9519f4d3555d20651509d8"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.595836 2753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:06.596618 2768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.596788 2753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:06.596928 2753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "ef9c4e9c4eaa453a882d055a50b98a72"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "210a68820808a501b3551bfa239aba25"
server_key_iv: "6422a6b20a9519f4d3555d20651509d8"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.597013 2753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:06.617720 2753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:06.620357 2753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:06.620530 2753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:06.620786 2753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:06.621145 2753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:06.621204 2753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.621250 2753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:06.621274 2753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.627538 2753 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:45665
I20260928 06:23:06.627601 2881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:45665 every 8 connection(s)
I20260928 06:23:06.627979 2753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:23:06.628989 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 2753
I20260928 06:23:06.629107 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:06.629381 32601 external_mini_cluster.cc:1428] Setting key 0b2042a822228f2b997f31d009b0900f
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[2512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
I20260928 06:23:06.640133 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:06.629021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:59999 (local address 127.31.214.126:32807)
0928 06:23:06.629073 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:06.629075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:06.629415 (+ 340us) server_negotiation.cc:407] Connection header received
0928 06:23:06.630250 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:06.630251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:06.630264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:06.630331 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:06.631728 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.632541 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.633819 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.634105 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.634183 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:06.636906 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:06.636908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:06.636908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:06.636928 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:06.638473 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:06.638946 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:06.638948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:06.638948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:06.639031 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:06.639343 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:06.639344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:06.639344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:06.639478 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:06.639604 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:06.639933 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 06:23:06.640032 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:23:06.640390 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:06.629034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:32807 (local address 127.31.214.66:59999)
0928 06:23:06.629282 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:06.629321 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:06.630111 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:06.630428 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:06.630433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:06.631299 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:06.631599 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:06.631606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.632662 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.632663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:06.633722 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:06.633725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.634207 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.634208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:06.634365 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:06.635037 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:06.635049 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:06.636793 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:06.638586 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:06.638596 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:06.638599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:06.638860 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:06.639109 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:06.639110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:06.639110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:06.639259 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:06.639594 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:06.639603 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:06.639770 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 06:23:06.639911 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 06:23:06.640092 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":134,"threads_started":1}
I20260928 06:23:06.640916 2882 heartbeater.cc:347] Connected to a master server at 127.31.214.126:32807
I20260928 06:23:06.641062 2882 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:06.641348 2882 heartbeater.cc:511] Master 127.31.214.126:32807 requested a full tablet report, sending...
I20260928 06:23:06.641845 2557 ts_manager.cc:194] Registered new tserver with Master: ef9c4e9c4eaa453a882d055a50b98a72 (127.31.214.66:45665)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
I20260928 06:23:06.642349 2557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:59999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:23:06.661020 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:32807
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:06.747972 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:06.748184 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:06.748207 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:06.749444 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:06.749487 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:06.749513 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:06.750977 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:06.751179 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:06.751375 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:06.756763 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:06.756775 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:06.756937 2887 server_base.cc:1034] running on GCE node
W20260928 06:23:06.756743 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:06.757251 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:06.757563 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:06.758759 2887 hybrid_clock.cc:648] HybridClock initialized: now 1790576586758732 us; error 38 us; skew 500 ppm
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[2512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:06.761893 2887 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:23:06.762696 2887 webserver.cc:460] Webserver started at http://127.31.214.67:43447/ using document root <none> and password file <none>
I20260928 06:23:06.763010 2887 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:06.763077 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:06.763222 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:06.764415 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "520250340f7e4f58b48ddf5b6719c979"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "a32b550dfeb2c2e26d0e57b691a7a17e"
server_key_iv: "751ef49a7aab83a10d942daba0f3ac57"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.764822 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "520250340f7e4f58b48ddf5b6719c979"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "a32b550dfeb2c2e26d0e57b691a7a17e"
server_key_iv: "751ef49a7aab83a10d942daba0f3ac57"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.766361 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:23:06.767468 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.768105 2887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:06.768227 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "520250340f7e4f58b48ddf5b6719c979"
format_stamp: "Formatted at 2026-09-28 06:23:06 on dist-test-slave-q03f"
server_key: "a32b550dfeb2c2e26d0e57b691a7a17e"
server_key_iv: "751ef49a7aab83a10d942daba0f3ac57"
server_key_version: "encryptionkey@0"
I20260928 06:23:06.768316 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:06.783969 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:06.786425 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:06.786569 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:06.786839 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:06.787135 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:06.787168 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.787190 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:06.787202 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:06.793493 2887 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:41143
I20260928 06:23:06.793534 3015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:41143 every 8 connection(s)
I20260928 06:23:06.793879 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:06.797065 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 2887
I20260928 06:23:06.797145 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:06.797343 32601 external_mini_cluster.cc:1428] Setting key 89017f27d498e8c847247d9cbb8d8b54
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[2512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576586, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:06.804214 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:06.794665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:32807 (local address 127.31.214.67:48385)
0928 06:23:06.794849 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:06.794886 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:06.795403 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:06.795678 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:06.795682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:06.796285 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:06.796498 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:06.796503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.797269 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.797270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:06.798445 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:06.798448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.798923 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.798923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:06.799049 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:06.799619 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:06.799627 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:06.801163 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:06.802937 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:06.802943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:06.802945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:06.803163 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:06.803354 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:06.803354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:06.803355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:06.803450 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:06.803721 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:06.803725 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:06.803841 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 06:23:06.803923 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:23:06.804047 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260928 06:23:06.804245 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:06.794692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:48385 (local address 127.31.214.126:32807)
0928 06:23:06.794737 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:06.794739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:06.794992 (+ 253us) server_negotiation.cc:407] Connection header received
0928 06:23:06.795522 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:06.795523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:06.795536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:06.795598 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:06.796615 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.797157 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.798558 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:06.798841 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:06.798901 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:06.801259 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:06.801260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:06.801261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:06.801267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:06.802863 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:06.803232 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:06.803232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:06.803233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:06.803287 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:06.803559 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:06.803559 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:06.803560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:06.803667 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:06.803717 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:06.803951 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 06:23:06.804038 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:23:06.804683 3016 heartbeater.cc:347] Connected to a master server at 127.31.214.126:32807
I20260928 06:23:06.804797 3016 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:06.804996 3016 heartbeater.cc:511] Master 127.31.214.126:32807 requested a full tablet report, sending...
I20260928 06:23:06.805477 2557 ts_manager.cc:194] Registered new tserver with Master: 520250340f7e4f58b48ddf5b6719c979 (127.31.214.67:41143)
I20260928 06:23:06.805821 2557 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:48385
I20260928 06:23:06.809607 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:23:06.825950 2608 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:06.824862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.31.214.126:32807)
0928 06:23:06.824940 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:06.824943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:06.825040 (+ 97us) server_negotiation.cc:407] Connection header received
0928 06:23:06.825148 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:06.825149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:06.825159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:06.825221 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:06.825865 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38576: BlockingRecv error: recv got EOF from 127.0.0.1:38576 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:23:06.826862 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 2619
I20260928 06:23:06.831799 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 2753
I20260928 06:23:06.836369 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 2887
I20260928 06:23:06.841010 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 2525
2026-09-28T06:23:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1929 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[3028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[3028](info): set up 2 sockets
Sep 28 06:23:06 dist-test-slave-q03f krb5kdc[3028](info): commencing operation
krb5kdc: starting...
W20260928 06:23:07.889995 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:07 dist-test-slave-q03f krb5kdc[3028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:23:08.025167 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:43157
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:08.109043 3041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:08.109302 3041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:08.109342 3041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:08.110555 3041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:08.110628 3041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:08.110656 3041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:08.110677 3041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:08.110707 3041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:08.112254 3041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:43157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:08.112545 3041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:08.112784 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:08.117735 3049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:08.117769 3047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:08.117751 3046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:08.118088 3041 server_base.cc:1034] running on GCE node
I20260928 06:23:08.118295 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:08.118561 3041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:08.119715 3041 hybrid_clock.cc:648] HybridClock initialized: now 1790576588119705 us; error 35 us; skew 500 ppm
Sep 28 06:23:08 dist-test-slave-q03f krb5kdc[3028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576588, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:08.132674 3041 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:23:08.133267 3041 webserver.cc:460] Webserver started at http://127.31.214.126:37437/ using document root <none> and password file <none>
I20260928 06:23:08.133508 3041 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:08.133554 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:08.133654 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:08.134819 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "84c228b78b094a95b9de09a2d22c65d2"
format_stamp: "Formatted at 2026-09-28 06:23:08 on dist-test-slave-q03f"
server_key: "0a0be6b86d69b018e165750c202849a0"
server_key_iv: "d72fd6add8db2e8c75715922cd2322d4"
server_key_version: "encryptionkey@0"
I20260928 06:23:08.135145 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "84c228b78b094a95b9de09a2d22c65d2"
format_stamp: "Formatted at 2026-09-28 06:23:08 on dist-test-slave-q03f"
server_key: "0a0be6b86d69b018e165750c202849a0"
server_key_iv: "d72fd6add8db2e8c75715922cd2322d4"
server_key_version: "encryptionkey@0"
I20260928 06:23:08.136644 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:23:08.137547 3056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:08.137750 3041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:08.137858 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "84c228b78b094a95b9de09a2d22c65d2"
format_stamp: "Formatted at 2026-09-28 06:23:08 on dist-test-slave-q03f"
server_key: "0a0be6b86d69b018e165750c202849a0"
server_key_iv: "d72fd6add8db2e8c75715922cd2322d4"
server_key_version: "encryptionkey@0"
I20260928 06:23:08.137923 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:08.154218 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:08.156706 3041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:08.156831 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:08.161001 3041 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:43157
I20260928 06:23:08.161360 3108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:43157 every 8 connection(s)
I20260928 06:23:08.161442 3041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:08.162365 3109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:08.164945 3109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2: Bootstrap starting.
I20260928 06:23:08.165652 3109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:08.165989 3109 log.cc:824] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:08.166785 3109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2: No bootstrap required, opened a new log
I20260928 06:23:08.168560 3109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c228b78b094a95b9de09a2d22c65d2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43157 } }
I20260928 06:23:08.168737 3109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:08.168779 3109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c228b78b094a95b9de09a2d22c65d2, State: Initialized, Role: FOLLOWER
I20260928 06:23:08.168980 3109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c228b78b094a95b9de09a2d22c65d2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43157 } }
I20260928 06:23:08.169085 3109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:08.169130 3109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:08.169183 3109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:08.170233 3109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c228b78b094a95b9de09a2d22c65d2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43157 } }
I20260928 06:23:08.170367 3109 leader_election.cc:302] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c228b78b094a95b9de09a2d22c65d2; no voters:
I20260928 06:23:08.170586 3109 leader_election.cc:288] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:08.170650 3112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:08.170852 3112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [term 1 LEADER]: Becoming Leader. State: Replica: 84c228b78b094a95b9de09a2d22c65d2, State: Running, Role: LEADER
I20260928 06:23:08.170895 3109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:08.170955 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 3041
I20260928 06:23:08.170964 3112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c228b78b094a95b9de09a2d22c65d2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43157 } }
I20260928 06:23:08.171046 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:08.171268 32601 external_mini_cluster.cc:1428] Setting key 2021cc9247439a32cb4f5f260a02638a
I20260928 06:23:08.171295 3112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c228b78b094a95b9de09a2d22c65d2. Latest consensus state: current_term: 1 leader_uuid: "84c228b78b094a95b9de09a2d22c65d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c228b78b094a95b9de09a2d22c65d2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43157 } } }
I20260928 06:23:08.171350 3114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84c228b78b094a95b9de09a2d22c65d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c228b78b094a95b9de09a2d22c65d2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43157 } } }
I20260928 06:23:08.171379 3112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:08.171578 3114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:08.171808 3119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:08.172408 3119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:08.174319 3119 catalog_manager.cc:1347] Generated new cluster ID: 54e8544f07aa46fb9fe3b6d5a74a583a
I20260928 06:23:08.174376 3119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:23:08 dist-test-slave-q03f krb5kdc[3028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:08.184211 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:08.171956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.31.214.126:43157)
0928 06:23:08.172475 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:08.172486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:08.172773 (+ 287us) server_negotiation.cc:407] Connection header received
0928 06:23:08.172917 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:08.172924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:08.172971 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:08.173114 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:08.173982 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:08.175415 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:08.176492 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:08.177178 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:08.177296 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:08.180238 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:08.180243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:08.180244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:08.180258 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:08.182468 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:08.182901 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:08.182904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:08.182904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:08.183019 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:08.183237 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:08.183238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:08.183238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:08.183559 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:08.183710 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:08.183826 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:23:08.183967 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":82,"threads_started":1}
I20260928 06:23:08.185225 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:23:08.194479 3119 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:08.195055 3119 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:08.199393 3119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84c228b78b094a95b9de09a2d22c65d2: Generated new TSK 0
I20260928 06:23:08.199707 3119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:23:08.204113 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:08.200176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.31.214.126:43157)
0928 06:23:08.200239 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:08.200241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:08.200251 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:23:08.200278 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:08.200279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:08.200291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:08.200349 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:08.200854 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:08.201358 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:08.202579 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:08.202810 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:08.202875 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:08.203616 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:08.203618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:08.203618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:08.203842 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:08.203859 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:08.203866 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:08.203914 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:08.203999 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:23:08.204024 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:23:08.206950 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 3041
2026-09-28T06:23:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1362 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:08 dist-test-slave-q03f krb5kdc[3142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:08 dist-test-slave-q03f krb5kdc[3142](info): set up 2 sockets
Sep 28 06:23:08 dist-test-slave-q03f krb5kdc[3142](info): commencing operation
krb5kdc: starting...
W20260928 06:23:09.250531 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:09 dist-test-slave-q03f krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:23:09.378067 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:41637
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:41637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:09.463111 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:09.463336 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:09.463361 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:09.464627 3155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:09.464676 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:09.464696 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:09.464711 3155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:09.464723 3155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:09.466410 3155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:41637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:41637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:09.466707 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:09.466960 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:09.472218 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:09.472155 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:09.472250 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:09.472572 3155 server_base.cc:1034] running on GCE node
I20260928 06:23:09.472857 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:09.473197 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:09.474349 3155 hybrid_clock.cc:648] HybridClock initialized: now 1790576589474326 us; error 38 us; skew 500 ppm
Sep 28 06:23:09 dist-test-slave-q03f krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576589, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:09.477018 3155 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:23:09.477509 3155 webserver.cc:460] Webserver started at http://127.31.214.126:46443/ using document root <none> and password file <none>
I20260928 06:23:09.477761 3155 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:09.477814 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:09.478003 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:09.479141 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "16d7943c58894acdaa2bf9e0f22306be"
format_stamp: "Formatted at 2026-09-28 06:23:09 on dist-test-slave-q03f"
server_key: "31cb39f3aef1c5d791283fca9d46f59d"
server_key_iv: "2333aed748bcfba24cb27c1cfe7f006b"
server_key_version: "encryptionkey@0"
I20260928 06:23:09.479513 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "16d7943c58894acdaa2bf9e0f22306be"
format_stamp: "Formatted at 2026-09-28 06:23:09 on dist-test-slave-q03f"
server_key: "31cb39f3aef1c5d791283fca9d46f59d"
server_key_iv: "2333aed748bcfba24cb27c1cfe7f006b"
server_key_version: "encryptionkey@0"
I20260928 06:23:09.480976 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:23:09.482030 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:09.482237 3155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:09.482440 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "16d7943c58894acdaa2bf9e0f22306be"
format_stamp: "Formatted at 2026-09-28 06:23:09 on dist-test-slave-q03f"
server_key: "31cb39f3aef1c5d791283fca9d46f59d"
server_key_iv: "2333aed748bcfba24cb27c1cfe7f006b"
server_key_version: "encryptionkey@0"
I20260928 06:23:09.482525 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:09.500010 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:09.502633 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:09.502832 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:09.507607 3155 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:41637
I20260928 06:23:09.507668 3222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:41637 every 8 connection(s)
I20260928 06:23:09.508091 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:09.509168 3223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:09.511569 3223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be: Bootstrap starting.
I20260928 06:23:09.512189 3223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:09.512493 3223 log.cc:824] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:09.513240 3223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be: No bootstrap required, opened a new log
I20260928 06:23:09.513680 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 3155
I20260928 06:23:09.513813 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:09.514079 32601 external_mini_cluster.cc:1428] Setting key 1be113d984dbeffdbb0215e0b76cdfb7
I20260928 06:23:09.515672 3223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d7943c58894acdaa2bf9e0f22306be" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41637 } }
I20260928 06:23:09.515856 3223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:09.515897 3223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d7943c58894acdaa2bf9e0f22306be, State: Initialized, Role: FOLLOWER
I20260928 06:23:09.516021 3223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d7943c58894acdaa2bf9e0f22306be" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41637 } }
I20260928 06:23:09.516103 3223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:09.516125 3223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:09.516194 3223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:09.517280 3223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d7943c58894acdaa2bf9e0f22306be" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41637 } }
I20260928 06:23:09.517388 3223 leader_election.cc:302] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d7943c58894acdaa2bf9e0f22306be; no voters:
I20260928 06:23:09.517575 3223 leader_election.cc:288] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:09.517632 3228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:09.517794 3228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [term 1 LEADER]: Becoming Leader. State: Replica: 16d7943c58894acdaa2bf9e0f22306be, State: Running, Role: LEADER
I20260928 06:23:09.517836 3223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:09.517995 3228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d7943c58894acdaa2bf9e0f22306be" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41637 } }
I20260928 06:23:09.519606 3229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d7943c58894acdaa2bf9e0f22306be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d7943c58894acdaa2bf9e0f22306be" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41637 } } }
I20260928 06:23:09.519706 3229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:09.519892 3230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d7943c58894acdaa2bf9e0f22306be. Latest consensus state: current_term: 1 leader_uuid: "16d7943c58894acdaa2bf9e0f22306be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d7943c58894acdaa2bf9e0f22306be" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41637 } } }
I20260928 06:23:09.519961 3230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be [sys.catalog]: This master's current role is: LEADER
W20260928 06:23:09.520025 3243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:23:09.520078 3243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:23:09.520167 3244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:09.520336 3244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:23:09 dist-test-slave-q03f krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:09.522729 3244 catalog_manager.cc:1347] Generated new cluster ID: 1bec46a19194424185cbe5c518d7947b
I20260928 06:23:09.522779 3244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:09.527160 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:09.514636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.31.214.126:41637)
0928 06:23:09.514868 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:09.514878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:09.514920 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:23:09.515626 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:09.515636 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:09.515705 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:09.515864 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:09.516707 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.518200 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.519365 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.520044 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.520159 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:09.523198 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:09.523202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:09.523202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:09.523215 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:09.525305 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:09.525733 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:09.525736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:09.525736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:09.525887 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:09.526123 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:09.526125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:09.526125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:09.526466 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:09.526653 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:09.526735 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:23:09.526938 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
I20260928 06:23:09.531396 3244 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:09.531999 3244 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:09.547866 3244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16d7943c58894acdaa2bf9e0f22306be: Generated new TSK 0
I20260928 06:23:09.548110 3244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:23:09.587097 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:41637
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:09.693152 3249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:09.693513 3249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:09.693559 3249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:09.695324 3249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:09.695429 3249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:09.695521 3249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:09.697194 3249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:41637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:09.697508 3249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:09.697779 3249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:09.703403 3254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:09.703403 3257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:09.703405 3255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:09.703692 3249 server_base.cc:1034] running on GCE node
I20260928 06:23:09.703994 3249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:09.704304 3249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:09.705514 3249 hybrid_clock.cc:648] HybridClock initialized: now 1790576589705494 us; error 49 us; skew 500 ppm
Sep 28 06:23:09 dist-test-slave-q03f krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576589, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:09.708410 3249 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:23:09.708889 3249 webserver.cc:460] Webserver started at http://127.31.214.65:39517/ using document root <none> and password file <none>
I20260928 06:23:09.709122 3249 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:09.709188 3249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:09.709353 3249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:09.710531 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "c0b06ea2177048578c6f1748b6bd23cf"
format_stamp: "Formatted at 2026-09-28 06:23:09 on dist-test-slave-q03f"
server_key: "6c618d0e684dc5ea6268e3ab5979f166"
server_key_iv: "5424dc4cdc8f6216a471b9e05fef16b6"
server_key_version: "encryptionkey@0"
I20260928 06:23:09.710922 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "c0b06ea2177048578c6f1748b6bd23cf"
format_stamp: "Formatted at 2026-09-28 06:23:09 on dist-test-slave-q03f"
server_key: "6c618d0e684dc5ea6268e3ab5979f166"
server_key_iv: "5424dc4cdc8f6216a471b9e05fef16b6"
server_key_version: "encryptionkey@0"
I20260928 06:23:09.712371 3249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:23:09.713181 3264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:09.713369 3249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:09.713477 3249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "c0b06ea2177048578c6f1748b6bd23cf"
format_stamp: "Formatted at 2026-09-28 06:23:09 on dist-test-slave-q03f"
server_key: "6c618d0e684dc5ea6268e3ab5979f166"
server_key_iv: "5424dc4cdc8f6216a471b9e05fef16b6"
server_key_version: "encryptionkey@0"
I20260928 06:23:09.713553 3249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:09.727607 3249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:09.729696 3249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:09.729882 3249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:09.730152 3249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:09.730485 3249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:09.730540 3249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:09.730581 3249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:09.730625 3249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:09.736714 3249 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:40739
I20260928 06:23:09.736783 3377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:40739 every 8 connection(s)
I20260928 06:23:09.737164 3249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:23:09 dist-test-slave-q03f krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576589, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:09.744210 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 3249
I20260928 06:23:09.744338 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:23:09.744560 32601 external_mini_cluster.cc:1428] Setting key 464ba7244267efc04842c9817353db4c
I20260928 06:23:09.748116 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:09.737996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:45361 (local address 127.31.214.126:41637)
0928 06:23:09.738046 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:09.738048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:09.738279 (+ 231us) server_negotiation.cc:407] Connection header received
0928 06:23:09.738798 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:09.738800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:09.738862 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:09.738928 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:09.739960 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.740514 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.741568 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.741820 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.741919 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:09.744499 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:09.744501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:09.744501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:09.744512 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:09.746369 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:09.746919 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:09.746920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:09.746921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:09.746995 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:09.747304 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:09.747304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:09.747305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:09.747482 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:09.747556 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:09.747863 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 06:23:09.747966 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:23:09.748137 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:09.737972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:41637 (local address 127.31.214.65:45361)
0928 06:23:09.738156 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:09.738201 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:09.738704 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:09.739038 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:09.739042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:09.739645 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:09.739856 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:09.739861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.740609 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.740610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:09.741454 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:09.741456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.742001 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.742002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:09.742131 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:09.742759 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:09.742768 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:09.744362 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:09.746458 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:09.746470 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:09.746473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:09.746837 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:09.747114 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:09.747114 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:09.747115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:09.747235 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:09.747567 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:09.747574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:09.747709 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:23:09.747802 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 06:23:09.747956 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260928 06:23:09.748711 3378 heartbeater.cc:347] Connected to a master server at 127.31.214.126:41637
I20260928 06:23:09.748823 3378 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:09.749022 3378 heartbeater.cc:511] Master 127.31.214.126:41637 requested a full tablet report, sending...
I20260928 06:23:09.749765 3187 ts_manager.cc:194] Registered new tserver with Master: c0b06ea2177048578c6f1748b6bd23cf (127.31.214.65:40739)
I20260928 06:23:09.750838 3187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:45361
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:23:09.774504 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:41637
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:09.858120 3383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:09.858340 3383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:09.858364 3383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:09.859619 3383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:09.859668 3383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:09.859695 3383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:09.861230 3383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:41637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:09.861502 3383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:09.861754 3383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:09.867111 3388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:09.867193 3383 server_base.cc:1034] running on GCE node
W20260928 06:23:09.867122 3389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:09.867125 3391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:09.867508 3383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:09.867743 3383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:09.868902 3383 hybrid_clock.cc:648] HybridClock initialized: now 1790576589868885 us; error 33 us; skew 500 ppm
Sep 28 06:23:09 dist-test-slave-q03f krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576589, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:09.871726 3383 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:23:09.872176 3383 webserver.cc:460] Webserver started at http://127.31.214.66:39883/ using document root <none> and password file <none>
I20260928 06:23:09.872373 3383 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:09.872419 3383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:09.872519 3383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:09.873602 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "01b62c7df9514eafbbc02f782a7ef8b0"
format_stamp: "Formatted at 2026-09-28 06:23:09 on dist-test-slave-q03f"
server_key: "5b4580a570902613566a137905d7ba3b"
server_key_iv: "1f860ebe0bb841dc7a10dcd4fdc2a86d"
server_key_version: "encryptionkey@0"
I20260928 06:23:09.873929 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "01b62c7df9514eafbbc02f782a7ef8b0"
format_stamp: "Formatted at 2026-09-28 06:23:09 on dist-test-slave-q03f"
server_key: "5b4580a570902613566a137905d7ba3b"
server_key_iv: "1f860ebe0bb841dc7a10dcd4fdc2a86d"
server_key_version: "encryptionkey@0"
I20260928 06:23:09.875334 3383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:09.876104 3398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:09.876394 3383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:09.876482 3383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "01b62c7df9514eafbbc02f782a7ef8b0"
format_stamp: "Formatted at 2026-09-28 06:23:09 on dist-test-slave-q03f"
server_key: "5b4580a570902613566a137905d7ba3b"
server_key_iv: "1f860ebe0bb841dc7a10dcd4fdc2a86d"
server_key_version: "encryptionkey@0"
I20260928 06:23:09.876539 3383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:09.902791 3383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:09.905393 3383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:09.905609 3383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:09.905854 3383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:09.906303 3383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:09.906347 3383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:09.906404 3383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:09.906445 3383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:09.913072 3383 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:34309
I20260928 06:23:09.913125 3511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:34309 every 8 connection(s)
I20260928 06:23:09.913508 3383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:23:09.920204 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 3383
Sep 28 06:23:09 dist-test-slave-q03f krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576589, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:09.920324 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:09.920567 32601 external_mini_cluster.cc:1428] Setting key 716faa8f5aba0c397c4039532ffd9011
I20260928 06:23:09.923815 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:09.914330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:34373 (local address 127.31.214.126:41637)
0928 06:23:09.914402 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:09.914405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:09.914630 (+ 225us) server_negotiation.cc:407] Connection header received
0928 06:23:09.915184 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:09.915186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:09.915200 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:09.915245 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:09.916285 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.916875 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.918112 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.918317 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.918404 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:09.920819 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:09.920821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:09.920821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:09.920830 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:09.922393 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:09.922826 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:09.922827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:09.922828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:09.922888 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:09.923151 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:09.923152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:09.923153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:09.923277 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:09.923354 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:09.923630 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 06:23:09.923720 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:23:09.924036 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:09.914298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:41637 (local address 127.31.214.66:34373)
0928 06:23:09.914503 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:09.914544 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:09.915070 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:09.915314 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:09.915318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:09.915981 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:09.916180 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:09.916185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.916989 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.916990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:09.918006 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:09.918009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:09.918420 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:09.918421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:09.918576 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:09.919232 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:09.919240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:09.920725 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:09.922500 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:09.922507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:09.922510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:09.922746 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:09.922963 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:09.922964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:09.922965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:09.923084 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:09.923344 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:09.923351 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:09.923499 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 06:23:09.923610 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:23:09.923775 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260928 06:23:09.924621 3512 heartbeater.cc:347] Connected to a master server at 127.31.214.126:41637
I20260928 06:23:09.924737 3512 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:09.924968 3512 heartbeater.cc:511] Master 127.31.214.126:41637 requested a full tablet report, sending...
I20260928 06:23:09.925537 3187 ts_manager.cc:194] Registered new tserver with Master: 01b62c7df9514eafbbc02f782a7ef8b0 (127.31.214.66:34309)
I20260928 06:23:09.926149 3187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:34373
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:23:09.950294 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:41637
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:10.042595 3517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:10.042845 3517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:10.042876 3517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:10.044255 3517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:10.044322 3517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:10.044374 3517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:10.046205 3517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:41637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:10.046487 3517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:10.046772 3517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:10.052733 3523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:10.052788 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:10.053007 3522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:10.053074 3517 server_base.cc:1034] running on GCE node
I20260928 06:23:10.053326 3517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:10.053552 3517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:10.054728 3517 hybrid_clock.cc:648] HybridClock initialized: now 1790576590054706 us; error 33 us; skew 500 ppm
Sep 28 06:23:10 dist-test-slave-q03f krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576590, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:10.057659 3517 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:23:10.058218 3517 webserver.cc:460] Webserver started at http://127.31.214.67:33289/ using document root <none> and password file <none>
I20260928 06:23:10.058449 3517 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:10.058496 3517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:10.058595 3517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:10.059711 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "e1857d7fd41f46d1b946d1a4cd6db9e9"
format_stamp: "Formatted at 2026-09-28 06:23:10 on dist-test-slave-q03f"
server_key: "e56e50accb396d0898073e65ac9ce4e5"
server_key_iv: "28e7caf602130652c86c710b4b2374e7"
server_key_version: "encryptionkey@0"
I20260928 06:23:10.060050 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "e1857d7fd41f46d1b946d1a4cd6db9e9"
format_stamp: "Formatted at 2026-09-28 06:23:10 on dist-test-slave-q03f"
server_key: "e56e50accb396d0898073e65ac9ce4e5"
server_key_iv: "28e7caf602130652c86c710b4b2374e7"
server_key_version: "encryptionkey@0"
I20260928 06:23:10.061452 3517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:10.062366 3532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:10.062604 3517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:23:10.062691 3517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "e1857d7fd41f46d1b946d1a4cd6db9e9"
format_stamp: "Formatted at 2026-09-28 06:23:10 on dist-test-slave-q03f"
server_key: "e56e50accb396d0898073e65ac9ce4e5"
server_key_iv: "28e7caf602130652c86c710b4b2374e7"
server_key_version: "encryptionkey@0"
I20260928 06:23:10.062754 3517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:10.070483 3517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:10.072736 3517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:10.072852 3517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:10.073057 3517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:10.073381 3517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:10.073418 3517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:10.073441 3517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:10.073485 3517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:10.079852 3517 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:44091
I20260928 06:23:10.079950 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:44091 every 8 connection(s)
I20260928 06:23:10.080343 3517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:10.086117 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 3517
I20260928 06:23:10.086218 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:10.086508 32601 external_mini_cluster.cc:1428] Setting key cf447a86e1134722b22d144f86b6cecf
Sep 28 06:23:10 dist-test-slave-q03f krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576590, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:10.090972 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:10.081191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:36261 (local address 127.31.214.126:41637)
0928 06:23:10.081253 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:10.081255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:10.081474 (+ 219us) server_negotiation.cc:407] Connection header received
0928 06:23:10.082136 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:10.082137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:10.082151 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:10.082203 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:10.083228 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:10.083824 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:10.084887 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:10.085090 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:10.085178 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:10.087995 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:10.087997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:10.087997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:10.088005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:10.089411 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:10.089886 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:10.089887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:10.089887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:10.090000 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:10.090260 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:10.090261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:10.090262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:10.090406 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:10.090484 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:10.090773 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 06:23:10.090878 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:23:10.091176 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:10.081151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:41637 (local address 127.31.214.67:36261)
0928 06:23:10.081319 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:10.081371 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:10.082005 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:10.082282 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:10.082286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:10.082912 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:10.083119 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:10.083124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:10.083936 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:10.083938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:10.084783 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:10.084785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:10.085187 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:10.085188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:10.085351 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:10.086062 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:10.086075 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:10.087889 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:10.089521 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:10.089528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:10.089530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:10.089776 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:10.090072 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:10.090073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:10.090074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:10.090190 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:10.090485 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:10.090492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:10.090637 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 06:23:10.090750 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:23:10.090915 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 06:23:10.091732 3646 heartbeater.cc:347] Connected to a master server at 127.31.214.126:41637
I20260928 06:23:10.091845 3646 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:10.092036 3646 heartbeater.cc:511] Master 127.31.214.126:41637 requested a full tablet report, sending...
I20260928 06:23:10.092449 3187 ts_manager.cc:194] Registered new tserver with Master: e1857d7fd41f46d1b946d1a4cd6db9e9 (127.31.214.67:44091)
I20260928 06:23:10.092876 3187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:36261
I20260928 06:23:10.098775 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:23:10 dist-test-slave-q03f krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576590, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 06:23:10 dist-test-slave-q03f krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576590, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:10.124667 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:10.116385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.31.214.126:41637)
0928 06:23:10.116459 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:10.116462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:10.116577 (+ 115us) server_negotiation.cc:407] Connection header received
0928 06:23:10.116634 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:10.116635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:10.116644 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:10.116698 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:10.117478 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:10.118043 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:10.118862 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:10.119124 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:10.119194 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:10.121826 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:10.121828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:10.121828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:10.121836 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:10.123444 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:10.123778 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:10.123778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:10.123779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:10.123831 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:10.124008 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:10.124008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:10.124009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:10.124177 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:10.124281 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:10.124488 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 06:23:10.124558 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 06:23:10.124975 3187 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33648
I20260928 06:23:10.125524 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 3249
I20260928 06:23:10.130494 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 3383
I20260928 06:23:10.135385 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 3517
I20260928 06:23:10.140165 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 3155
2026-09-28T06:23:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1937 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:10 dist-test-slave-q03f krb5kdc[3658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:10 dist-test-slave-q03f krb5kdc[3658](info): set up 2 sockets
Sep 28 06:23:10 dist-test-slave-q03f krb5kdc[3658](info): commencing operation
krb5kdc: starting...
W20260928 06:23:11.190075 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:11 dist-test-slave-q03f krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:23:11.325378 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:44871
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:44871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:11.409372 3671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:11.409591 3671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:11.409614 3671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:11.411290 3671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:11.411357 3671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:11.411383 3671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:11.411403 3671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:11.411424 3671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:11.414142 3671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:44871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:44871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:11.414451 3671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:11.414739 3671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:11.420715 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:11.420783 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:11.420732 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:11.420869 3671 server_base.cc:1034] running on GCE node
I20260928 06:23:11.421196 3671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:11.421525 3671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:11.422704 3671 hybrid_clock.cc:648] HybridClock initialized: now 1790576591422696 us; error 42 us; skew 500 ppm
Sep 28 06:23:11 dist-test-slave-q03f krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576591, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:11.425545 3671 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:23:11.426163 3671 webserver.cc:460] Webserver started at http://127.31.214.126:34469/ using document root <none> and password file <none>
I20260928 06:23:11.426416 3671 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:11.426491 3671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:11.426662 3671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:11.427834 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "79cc4b22f5164dc6aae515972a486915"
format_stamp: "Formatted at 2026-09-28 06:23:11 on dist-test-slave-q03f"
server_key: "5ecce1f1d83898fe00db31fdb61f4754"
server_key_iv: "ea166a1622dd8f876e47b952c1f4a847"
server_key_version: "encryptionkey@0"
I20260928 06:23:11.428261 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "79cc4b22f5164dc6aae515972a486915"
format_stamp: "Formatted at 2026-09-28 06:23:11 on dist-test-slave-q03f"
server_key: "5ecce1f1d83898fe00db31fdb61f4754"
server_key_iv: "ea166a1622dd8f876e47b952c1f4a847"
server_key_version: "encryptionkey@0"
I20260928 06:23:11.429848 3671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:11.430754 3686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:11.431010 3671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:11.431134 3671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "79cc4b22f5164dc6aae515972a486915"
format_stamp: "Formatted at 2026-09-28 06:23:11 on dist-test-slave-q03f"
server_key: "5ecce1f1d83898fe00db31fdb61f4754"
server_key_iv: "ea166a1622dd8f876e47b952c1f4a847"
server_key_version: "encryptionkey@0"
I20260928 06:23:11.431286 3671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:11.463055 3671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:11.465385 3671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:11.465576 3671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:11.470160 3671 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:44871
I20260928 06:23:11.470196 3738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:44871 every 8 connection(s)
I20260928 06:23:11.470718 3671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:11.471251 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 3671
I20260928 06:23:11.471385 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:11.471642 32601 external_mini_cluster.cc:1428] Setting key 74e6cbdbf212b2d42af11bd79c356d7e
I20260928 06:23:11.471711 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:11.474785 3739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915: Bootstrap starting.
I20260928 06:23:11.475430 3739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:11.475745 3739 log.cc:824] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:11.476559 3739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915: No bootstrap required, opened a new log
Sep 28 06:23:11 dist-test-slave-q03f krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:11.478698 3739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc4b22f5164dc6aae515972a486915" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44871 } }
I20260928 06:23:11.478886 3739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:11.478940 3739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc4b22f5164dc6aae515972a486915, State: Initialized, Role: FOLLOWER
I20260928 06:23:11.479110 3739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc4b22f5164dc6aae515972a486915" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44871 } }
I20260928 06:23:11.479210 3739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:11.479264 3739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:11.479312 3739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:11.480032 3739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc4b22f5164dc6aae515972a486915" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44871 } }
I20260928 06:23:11.480173 3739 leader_election.cc:302] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cc4b22f5164dc6aae515972a486915; no voters:
I20260928 06:23:11.480361 3739 leader_election.cc:288] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:11.480397 3744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:11.480515 3744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [term 1 LEADER]: Becoming Leader. State: Replica: 79cc4b22f5164dc6aae515972a486915, State: Running, Role: LEADER
I20260928 06:23:11.480602 3744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc4b22f5164dc6aae515972a486915" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44871 } }
I20260928 06:23:11.480664 3739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:11.480962 3744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79cc4b22f5164dc6aae515972a486915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc4b22f5164dc6aae515972a486915" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44871 } } }
I20260928 06:23:11.481035 3744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:11.481159 3744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79cc4b22f5164dc6aae515972a486915. Latest consensus state: current_term: 1 leader_uuid: "79cc4b22f5164dc6aae515972a486915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cc4b22f5164dc6aae515972a486915" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44871 } } }
I20260928 06:23:11.481225 3744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915 [sys.catalog]: This master's current role is: LEADER
W20260928 06:23:11.482630 3760 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:23:11.482698 3760 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:23:11.482761 3749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:11.482976 3749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:11.483655 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:11.472254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.31.214.126:44871)
0928 06:23:11.472452 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:11.472463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:11.472500 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:23:11.473146 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:11.473152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:11.473203 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:11.473355 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:11.474077 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.475042 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.475934 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.476482 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.476616 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:11.479203 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:11.479208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:11.479208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:11.479235 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:11.481333 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:11.481836 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:11.481839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:11.481840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:11.482000 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:11.482287 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:11.482288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:11.482288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:11.482614 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:11.483175 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:11.483247 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:23:11.483401 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260928 06:23:11.484853 3749 catalog_manager.cc:1347] Generated new cluster ID: 26808943945e498dbd6e3402461466e9
I20260928 06:23:11.484929 3749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:11.499212 3749 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:11.499882 3749 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:11.504487 3749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79cc4b22f5164dc6aae515972a486915: Generated new TSK 0
I20260928 06:23:11.504751 3749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:23:11.545600 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:44871
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:11.631593 3765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:11.631810 3765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:11.631834 3765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:11.633051 3765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:11.633096 3765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:11.633123 3765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:11.634881 3765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:44871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:11.635149 3765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:11.635455 3765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:11.641327 3773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:11.641425 3771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:11.641371 3770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:11.641582 3765 server_base.cc:1034] running on GCE node
I20260928 06:23:11.641820 3765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:11.642115 3765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:11.643324 3765 hybrid_clock.cc:648] HybridClock initialized: now 1790576591643296 us; error 41 us; skew 500 ppm
Sep 28 06:23:11 dist-test-slave-q03f krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576591, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:11.646389 3765 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:23:11.646962 3765 webserver.cc:460] Webserver started at http://127.31.214.65:44209/ using document root <none> and password file <none>
I20260928 06:23:11.647236 3765 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:11.647321 3765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:11.647478 3765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:11.648599 3765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "1e1079c2d74047f6bdc9d352cf69f110"
format_stamp: "Formatted at 2026-09-28 06:23:11 on dist-test-slave-q03f"
server_key: "005de625ef3bd608d37ca1aae50dbc2d"
server_key_iv: "74801338338ab571c4fe63f1dd48941d"
server_key_version: "encryptionkey@0"
I20260928 06:23:11.648964 3765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "1e1079c2d74047f6bdc9d352cf69f110"
format_stamp: "Formatted at 2026-09-28 06:23:11 on dist-test-slave-q03f"
server_key: "005de625ef3bd608d37ca1aae50dbc2d"
server_key_iv: "74801338338ab571c4fe63f1dd48941d"
server_key_version: "encryptionkey@0"
I20260928 06:23:11.650544 3765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:11.651547 3780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:11.651782 3765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:11.651903 3765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "1e1079c2d74047f6bdc9d352cf69f110"
format_stamp: "Formatted at 2026-09-28 06:23:11 on dist-test-slave-q03f"
server_key: "005de625ef3bd608d37ca1aae50dbc2d"
server_key_iv: "74801338338ab571c4fe63f1dd48941d"
server_key_version: "encryptionkey@0"
I20260928 06:23:11.651986 3765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:11.676626 3765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:11.679199 3765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:11.679515 3765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:11.679790 3765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:11.680126 3765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:11.680181 3765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:11.680235 3765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:11.680266 3765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:11.686677 3765 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:42745
I20260928 06:23:11.686746 3893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:42745 every 8 connection(s)
I20260928 06:23:11.687109 3765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:23:11.691548 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 3765
I20260928 06:23:11.691653 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:23:11.691867 32601 external_mini_cluster.cc:1428] Setting key 2a77cc0fc511fc22f9568b80cf279607
Sep 28 06:23:11 dist-test-slave-q03f krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576591, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:11.699879 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:11.688082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:54703 (local address 127.31.214.126:44871)
0928 06:23:11.688150 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:11.688152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:11.688380 (+ 228us) server_negotiation.cc:407] Connection header received
0928 06:23:11.689393 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:11.689395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:11.689405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:11.689465 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:11.690190 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.690774 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.692500 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.692710 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.692773 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:11.695817 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:11.695820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:11.695821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:11.695830 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:11.697834 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:11.698546 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:11.698548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:11.698548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:11.698622 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:11.698999 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:11.699000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:11.699001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:11.699167 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:11.699244 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:11.699657 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 06:23:11.699767 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:23:11.699980 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:11.688016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44871 (local address 127.31.214.65:54703)
0928 06:23:11.688201 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:11.688242 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:11.689275 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:11.689523 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:11.689527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:11.689786 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:11.690088 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:11.690093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.690889 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.690890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:11.692192 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:11.692195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.692800 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.692801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:11.692965 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:11.693565 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:11.693575 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:11.695661 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:11.698047 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:11.698057 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:11.698059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:11.698458 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:11.698747 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:11.698747 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:11.698748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:11.698887 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:11.699310 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:11.699318 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:11.699466 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 06:23:11.699585 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:23:11.699748 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260928 06:23:11.700585 3894 heartbeater.cc:347] Connected to a master server at 127.31.214.126:44871
I20260928 06:23:11.700713 3894 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:11.700970 3894 heartbeater.cc:511] Master 127.31.214.126:44871 requested a full tablet report, sending...
I20260928 06:23:11.701618 3703 ts_manager.cc:194] Registered new tserver with Master: 1e1079c2d74047f6bdc9d352cf69f110 (127.31.214.65:42745)
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
I20260928 06:23:11.702656 3703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:54703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:23:11.721478 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:44871
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:11.808161 3899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:11.808393 3899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:11.808419 3899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:11.809631 3899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:11.809681 3899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:11.809715 3899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:11.811372 3899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:44871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:11.811635 3899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:11.811918 3899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:11.817646 3904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:11.817688 3905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:11.817646 3907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:11.818344 3899 server_base.cc:1034] running on GCE node
I20260928 06:23:11.818712 3899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:11.819052 3899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:11.820253 3899 hybrid_clock.cc:648] HybridClock initialized: now 1790576591820238 us; error 61 us; skew 500 ppm
Sep 28 06:23:11 dist-test-slave-q03f krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576591, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:11.823150 3899 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:23:11.823640 3899 webserver.cc:460] Webserver started at http://127.31.214.66:44621/ using document root <none> and password file <none>
I20260928 06:23:11.823899 3899 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:11.823972 3899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:11.824126 3899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:11.825273 3899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "588fe812446541d7b9df8b936d725cbd"
format_stamp: "Formatted at 2026-09-28 06:23:11 on dist-test-slave-q03f"
server_key: "cb7561eba7b3bfa9f0fed69c17e96c5c"
server_key_iv: "87f3da2dc6bd4c2df51bdcebe4254ed6"
server_key_version: "encryptionkey@0"
I20260928 06:23:11.825651 3899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "588fe812446541d7b9df8b936d725cbd"
format_stamp: "Formatted at 2026-09-28 06:23:11 on dist-test-slave-q03f"
server_key: "cb7561eba7b3bfa9f0fed69c17e96c5c"
server_key_iv: "87f3da2dc6bd4c2df51bdcebe4254ed6"
server_key_version: "encryptionkey@0"
I20260928 06:23:11.827214 3899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260928 06:23:11.828033 3914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:11.828207 3899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:11.828313 3899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "588fe812446541d7b9df8b936d725cbd"
format_stamp: "Formatted at 2026-09-28 06:23:11 on dist-test-slave-q03f"
server_key: "cb7561eba7b3bfa9f0fed69c17e96c5c"
server_key_iv: "87f3da2dc6bd4c2df51bdcebe4254ed6"
server_key_version: "encryptionkey@0"
I20260928 06:23:11.828387 3899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:11.858306 3899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:11.861045 3899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:11.861243 3899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:11.861524 3899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:11.861871 3899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:11.861929 3899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:11.862037 3899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:11.862069 3899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:11.868417 3899 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:43103
I20260928 06:23:11.868510 4027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:43103 every 8 connection(s)
I20260928 06:23:11.868909 3899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:23:11 dist-test-slave-q03f krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576591, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:11.877538 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 3899
I20260928 06:23:11.877666 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:11.877928 32601 external_mini_cluster.cc:1428] Setting key e15f4bc18d999583dad4fcb63dc34676
I20260928 06:23:11.879820 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:11.869717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44871 (local address 127.31.214.66:34143)
0928 06:23:11.869919 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:11.870012 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:11.870969 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:11.871263 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:11.871267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:11.871545 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:11.871743 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:11.871747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.872758 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.872759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:11.873630 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:11.873633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.874111 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.874112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:11.874231 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:11.874776 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:11.874785 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:11.876295 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:11.878181 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:11.878190 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:11.878192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:11.878537 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:11.878842 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:11.878842 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:11.878843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:11.878974 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:11.879291 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:11.879295 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:11.879428 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 06:23:11.879514 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:23:11.879643 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260928 06:23:11.879838 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:11.869736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:34143 (local address 127.31.214.126:44871)
0928 06:23:11.869826 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:11.869829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:11.870100 (+ 271us) server_negotiation.cc:407] Connection header received
0928 06:23:11.871107 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:11.871109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:11.871120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:11.871182 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:11.871844 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.872614 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.873732 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:11.874015 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:11.874092 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:11.876419 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:11.876421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:11.876422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:11.876433 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:11.878077 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:11.878660 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:11.878661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:11.878662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:11.878740 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:11.879047 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:11.879048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:11.879048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:11.879219 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:11.879319 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:11.879570 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 06:23:11.879681 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:23:11.880689 4028 heartbeater.cc:347] Connected to a master server at 127.31.214.126:44871
I20260928 06:23:11.880820 4028 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:11.881033 4028 heartbeater.cc:511] Master 127.31.214.126:44871 requested a full tablet report, sending...
I20260928 06:23:11.881525 3703 ts_manager.cc:194] Registered new tserver with Master: 588fe812446541d7b9df8b936d725cbd (127.31.214.66:43103)
I20260928 06:23:11.882035 3703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:34143
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:23:11.906047 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:44871
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:11.990727 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:11.991016 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:11.991081 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:11.992314 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:11.992388 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:11.992434 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:11.994196 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:44871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:11.994493 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:11.994728 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:12.000370 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:12.000380 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:12.000380 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:12.000895 4033 server_base.cc:1034] running on GCE node
I20260928 06:23:12.001127 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:12.001441 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:12.002632 4033 hybrid_clock.cc:648] HybridClock initialized: now 1790576592002604 us; error 43 us; skew 500 ppm
Sep 28 06:23:11 dist-test-slave-q03f krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576591, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:12.006222 4033 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:23:12.006814 4033 webserver.cc:460] Webserver started at http://127.31.214.67:38307/ using document root <none> and password file <none>
I20260928 06:23:12.007117 4033 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:12.007184 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:12.007316 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:12.008731 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "be38072d0d8b4fd9848b39541c654c0e"
format_stamp: "Formatted at 2026-09-28 06:23:12 on dist-test-slave-q03f"
server_key: "cdc7925875e4e94834d8405d22bd9af1"
server_key_iv: "375336094aad41db71acf705d6936093"
server_key_version: "encryptionkey@0"
I20260928 06:23:12.009131 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "be38072d0d8b4fd9848b39541c654c0e"
format_stamp: "Formatted at 2026-09-28 06:23:12 on dist-test-slave-q03f"
server_key: "cdc7925875e4e94834d8405d22bd9af1"
server_key_iv: "375336094aad41db71acf705d6936093"
server_key_version: "encryptionkey@0"
I20260928 06:23:12.010674 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 06:23:12.011519 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:12.011714 4033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:12.011839 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "be38072d0d8b4fd9848b39541c654c0e"
format_stamp: "Formatted at 2026-09-28 06:23:12 on dist-test-slave-q03f"
server_key: "cdc7925875e4e94834d8405d22bd9af1"
server_key_iv: "375336094aad41db71acf705d6936093"
server_key_version: "encryptionkey@0"
I20260928 06:23:12.011917 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:12.037531 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:12.040714 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:12.040910 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:12.041220 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:12.041649 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:12.041709 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:12.041757 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:12.041796 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:12.048576 4033 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:35259
I20260928 06:23:12.048753 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:35259 every 8 connection(s)
I20260928 06:23:12.049063 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:12.051892 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 4033
I20260928 06:23:12.052007 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:12.052209 32601 external_mini_cluster.cc:1428] Setting key e7edb8725fcec3621ef26a770897b0db
Sep 28 06:23:12 dist-test-slave-q03f krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576591, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:12.060545 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:12.049907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:33095 (local address 127.31.214.126:44871)
0928 06:23:12.050007 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:12.050009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:12.050208 (+ 199us) server_negotiation.cc:407] Connection header received
0928 06:23:12.051232 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:12.051234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:12.051245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:12.051315 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:12.052469 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:12.053163 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:12.054420 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:12.054632 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:12.054700 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:12.057116 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:12.057118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:12.057118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:12.057126 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:12.058761 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:12.059343 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:12.059345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:12.059345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:12.059425 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:12.059798 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:12.059800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:12.059800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:12.059931 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:12.060004 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:12.060373 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 06:23:12.060470 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:23:12.060698 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:12.049829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44871 (local address 127.31.214.67:33095)
0928 06:23:12.050063 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:12.050112 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:12.051111 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:12.051413 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:12.051417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:12.051918 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:12.052259 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:12.052279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:12.053272 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:12.053273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:12.054285 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:12.054287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:12.054754 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:12.054756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:12.054886 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:12.055502 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:12.055510 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:12.057003 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:12.058865 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:12.058872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:12.058874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:12.059261 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:12.059513 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:12.059514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:12.059514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:12.059658 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:12.060023 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:12.060030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:12.060169 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 06:23:12.060264 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:23:12.060467 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260928 06:23:12.061342 4162 heartbeater.cc:347] Connected to a master server at 127.31.214.126:44871
I20260928 06:23:12.061462 4162 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:12.061722 4162 heartbeater.cc:511] Master 127.31.214.126:44871 requested a full tablet report, sending...
I20260928 06:23:12.062222 3703 ts_manager.cc:194] Registered new tserver with Master: be38072d0d8b4fd9848b39541c654c0e (127.31.214.67:35259)
I20260928 06:23:12.062628 3703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:33095
I20260928 06:23:12.064308 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:23:12 dist-test-slave-q03f krb5kdc[3658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:23:12 dist-test-slave-q03f krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:23:12.090358 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:12.081237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.31.214.65:42745)
0928 06:23:12.081402 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:12.081409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:12.081436 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:23:12.081470 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:12.081472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:12.081486 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:12.081559 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:12.082444 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:12.083254 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:12.084038 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:12.084414 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:12.084487 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:12.087104 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:12.087113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:12.087114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:12.087123 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:12.088901 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:12.089449 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:12.089452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:12.089453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:12.089527 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:12.089768 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:12.089768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:12.089769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:12.089917 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:12.090124 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:12.090209 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:23:12.090276 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
Sep 28 06:23:12 dist-test-slave-q03f krb5kdc[3658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:12.099815 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:12.092022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.31.214.126:44871)
0928 06:23:12.092140 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:12.092142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:12.092162 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:23:12.092434 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:12.092435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:12.092446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:12.092504 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:12.093040 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:12.093614 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:12.094494 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:12.094685 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:12.094743 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:12.097124 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:12.097125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:12.097126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:12.097132 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:12.098641 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:12.099009 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:12.099010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:12.099010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:12.099064 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:12.099232 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:12.099233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:12.099233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:12.099382 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:12.099464 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:12.099658 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:23:12.099723 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 06:23:12.100147 3703 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49478
I20260928 06:23:12.100745 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 3765
I20260928 06:23:12.105880 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 3899
I20260928 06:23:12.110903 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 4033
I20260928 06:23:12.115887 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 3671
2026-09-28T06:23:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1973 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:12 dist-test-slave-q03f krb5kdc[4174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:12 dist-test-slave-q03f krb5kdc[4174](info): set up 2 sockets
Sep 28 06:23:12 dist-test-slave-q03f krb5kdc[4174](info): commencing operation
krb5kdc: starting...
W20260928 06:23:13.262331 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:13 dist-test-slave-q03f krb5kdc[4174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:23:13.390476 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:37641
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:37641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:13.476109 4187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:13.476331 4187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:13.476357 4187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:13.477468 4187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:13.477514 4187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:13.477530 4187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:13.477543 4187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:13.477557 4187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:13.479125 4187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:37641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:37641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:13.479344 4187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:13.479538 4187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:13.484809 4195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:13.484856 4187 server_base.cc:1034] running on GCE node
W20260928 06:23:13.484818 4193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:13.484830 4192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:13.485369 4187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:13.485702 4187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:13.486912 4187 hybrid_clock.cc:648] HybridClock initialized: now 1790576593486889 us; error 61 us; skew 500 ppm
Sep 28 06:23:13 dist-test-slave-q03f krb5kdc[4174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576593, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:13.497114 4187 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:23:13.497596 4187 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:13.497685 4187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:13.497852 4187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:13.499078 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "d6318b5c29cc4bf5ac209dc1baa2d541"
format_stamp: "Formatted at 2026-09-28 06:23:13 on dist-test-slave-q03f"
server_key: "c4fb181c544b8ffe4320a2fbec000094"
server_key_iv: "3b1087e36b19c23e83bdde8c9589489b"
server_key_version: "encryptionkey@0"
I20260928 06:23:13.499480 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "d6318b5c29cc4bf5ac209dc1baa2d541"
format_stamp: "Formatted at 2026-09-28 06:23:13 on dist-test-slave-q03f"
server_key: "c4fb181c544b8ffe4320a2fbec000094"
server_key_iv: "3b1087e36b19c23e83bdde8c9589489b"
server_key_version: "encryptionkey@0"
I20260928 06:23:13.501020 4187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:13.501929 4201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:13.502195 4187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:13.502306 4187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "d6318b5c29cc4bf5ac209dc1baa2d541"
format_stamp: "Formatted at 2026-09-28 06:23:13 on dist-test-slave-q03f"
server_key: "c4fb181c544b8ffe4320a2fbec000094"
server_key_iv: "3b1087e36b19c23e83bdde8c9589489b"
server_key_version: "encryptionkey@0"
I20260928 06:23:13.502382 4187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:13.514472 4187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:13.516871 4187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:13.517084 4187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:13.522020 4187 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:37641
I20260928 06:23:13.522210 4253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:37641 every 8 connection(s)
I20260928 06:23:13.522446 4187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:13.523506 4254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:13.525817 4254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541: Bootstrap starting.
I20260928 06:23:13.526178 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 4187
I20260928 06:23:13.526309 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:13.526494 32601 external_mini_cluster.cc:1428] Setting key eed132367e61a5d4690a88d1c62a2abe
I20260928 06:23:13.526676 4254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:13.527048 4254 log.cc:824] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:13.527890 4254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541: No bootstrap required, opened a new log
I20260928 06:23:13.530318 4254 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6318b5c29cc4bf5ac209dc1baa2d541" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37641 } }
I20260928 06:23:13.530527 4254 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:13.530562 4254 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6318b5c29cc4bf5ac209dc1baa2d541, State: Initialized, Role: FOLLOWER
I20260928 06:23:13.530697 4254 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6318b5c29cc4bf5ac209dc1baa2d541" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37641 } }
I20260928 06:23:13.530768 4254 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:13.530797 4254 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:13.530843 4254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:13.531481 4254 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6318b5c29cc4bf5ac209dc1baa2d541" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37641 } }
I20260928 06:23:13.531628 4254 leader_election.cc:302] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6318b5c29cc4bf5ac209dc1baa2d541; no voters:
I20260928 06:23:13.531823 4254 leader_election.cc:288] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:13.531914 4259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:13.532068 4259 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [term 1 LEADER]: Becoming Leader. State: Replica: d6318b5c29cc4bf5ac209dc1baa2d541, State: Running, Role: LEADER
I20260928 06:23:13.532128 4254 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:13.532186 4259 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6318b5c29cc4bf5ac209dc1baa2d541" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37641 } }
I20260928 06:23:13.532521 4261 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6318b5c29cc4bf5ac209dc1baa2d541. Latest consensus state: current_term: 1 leader_uuid: "d6318b5c29cc4bf5ac209dc1baa2d541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6318b5c29cc4bf5ac209dc1baa2d541" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37641 } } }
I20260928 06:23:13.532621 4261 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:13.532824 4265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:13.533391 4265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:13.534042 4260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6318b5c29cc4bf5ac209dc1baa2d541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6318b5c29cc4bf5ac209dc1baa2d541" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37641 } } }
I20260928 06:23:13.534125 4260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541 [sys.catalog]: This master's current role is: LEADER
Sep 28 06:23:13 dist-test-slave-q03f krb5kdc[4174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:13.535434 4265 catalog_manager.cc:1347] Generated new cluster ID: ca3f90a35dcf4ac6a37a4accceb7de4d
I20260928 06:23:13.535488 4265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:13.541532 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:13.527123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.31.214.126:37641)
0928 06:23:13.527310 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:13.527320 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:13.527357 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:23:13.527890 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:13.527898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:13.527987 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:13.528123 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:13.528904 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.529821 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.530964 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.531456 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.531550 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:13.534736 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:13.534740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:13.534740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:13.534753 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:13.539660 (+ 4907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:13.540187 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:13.540191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:13.540191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:13.540310 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:13.540556 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:13.540557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:13.540558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:13.540878 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:13.541037 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:13.541182 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:23:13.541320 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260928 06:23:13.554049 4265 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:13.554692 4265 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:13.560380 4265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541: Generated new TSK 0
I20260928 06:23:13.560609 4265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:23:13.593812 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:37641
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:13.681583 4280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:13.681810 4280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:13.681839 4280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:13.683185 4280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:13.683241 4280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:13.683274 4280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:13.685025 4280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:37641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:13.685330 4280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:13.685591 4280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:13.691344 4285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:13.691327 4288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:13.691327 4286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:13.691635 4280 server_base.cc:1034] running on GCE node
I20260928 06:23:13.691874 4280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:13.692094 4280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:13.693297 4280 hybrid_clock.cc:648] HybridClock initialized: now 1790576593693227 us; error 87 us; skew 500 ppm
Sep 28 06:23:13 dist-test-slave-q03f krb5kdc[4174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576593, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:13.696283 4280 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:23:13.696759 4280 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:13.696810 4280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:13.696969 4280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:13.698164 4280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "7c8804fa75084ed297c97825bf2537ce"
format_stamp: "Formatted at 2026-09-28 06:23:13 on dist-test-slave-q03f"
server_key: "a6e2cc14b79011f6b90a1e1aa4ba7a1b"
server_key_iv: "f2985e87bca1660ef6690b23a8881a93"
server_key_version: "encryptionkey@0"
I20260928 06:23:13.698536 4280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "7c8804fa75084ed297c97825bf2537ce"
format_stamp: "Formatted at 2026-09-28 06:23:13 on dist-test-slave-q03f"
server_key: "a6e2cc14b79011f6b90a1e1aa4ba7a1b"
server_key_iv: "f2985e87bca1660ef6690b23a8881a93"
server_key_version: "encryptionkey@0"
I20260928 06:23:13.699946 4280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:23:13.700841 4294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:13.701074 4280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:13.701179 4280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "7c8804fa75084ed297c97825bf2537ce"
format_stamp: "Formatted at 2026-09-28 06:23:13 on dist-test-slave-q03f"
server_key: "a6e2cc14b79011f6b90a1e1aa4ba7a1b"
server_key_iv: "f2985e87bca1660ef6690b23a8881a93"
server_key_version: "encryptionkey@0"
I20260928 06:23:13.701259 4280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:13.723220 4280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:13.726027 4280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:13.726214 4280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:13.726503 4280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:13.726872 4280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:13.726934 4280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:13.726982 4280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:13.727015 4280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:13.733762 4280 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:42337
I20260928 06:23:13.733841 4407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:42337 every 8 connection(s)
I20260928 06:23:13.734174 4280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:23:13.739400 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 4280
I20260928 06:23:13.739509 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:23:13.739722 32601 external_mini_cluster.cc:1428] Setting key 8cc8e63e9dba3bdc932034308e905031
Sep 28 06:23:13 dist-test-slave-q03f krb5kdc[4174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576593, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:13.745338 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:13.735023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:53929 (local address 127.31.214.126:37641)
0928 06:23:13.735093 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:13.735095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:13.735346 (+ 251us) server_negotiation.cc:407] Connection header received
0928 06:23:13.735947 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:13.735948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:13.735962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:13.736027 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:13.737107 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.737682 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.738883 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.739131 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.739221 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:13.742153 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:13.742155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:13.742155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:13.742163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:13.743711 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:13.744273 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:13.744274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:13.744275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:13.744333 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:13.744630 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:13.744632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:13.744632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:13.744759 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:13.744838 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:13.745144 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 06:23:13.745239 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:23:13.745580 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:13.734995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37641 (local address 127.31.214.65:53929)
0928 06:23:13.735198 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:13.735238 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:13.735820 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:13.736132 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:13.736136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:13.736427 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:13.736977 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:13.736983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.737795 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.737797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:13.738763 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:13.738765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.739230 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.739231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:13.739374 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:13.740100 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:13.740111 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:13.742042 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:13.743802 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:13.743810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:13.743813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:13.744185 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:13.744409 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:13.744410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:13.744410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:13.744548 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:13.744829 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:13.744836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:13.744996 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 06:23:13.745110 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:23:13.745305 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":92,"threads_started":1}
I20260928 06:23:13.746273 4408 heartbeater.cc:347] Connected to a master server at 127.31.214.126:37641
I20260928 06:23:13.746416 4408 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:13.746626 4408 heartbeater.cc:511] Master 127.31.214.126:37641 requested a full tablet report, sending...
I20260928 06:23:13.747385 4218 ts_manager.cc:194] Registered new tserver with Master: 7c8804fa75084ed297c97825bf2537ce (127.31.214.65:42337)
I20260928 06:23:13.748160 4218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:53929
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:23:13.769840 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:37641
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:13.859494 4413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:13.859720 4413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:13.859746 4413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:13.860909 4413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:13.860951 4413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:13.860984 4413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:13.862538 4413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:37641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:13.862746 4413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:13.862951 4413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:13.868650 4421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:13.868649 4418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:13.868644 4419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:13.869084 4413 server_base.cc:1034] running on GCE node
I20260928 06:23:13.869375 4413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:13.869663 4413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:13.870844 4413 hybrid_clock.cc:648] HybridClock initialized: now 1790576593870828 us; error 52 us; skew 500 ppm
Sep 28 06:23:13 dist-test-slave-q03f krb5kdc[4174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576593, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:13.873739 4413 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:23:13.874362 4413 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:13.874456 4413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:13.874634 4413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:13.875918 4413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "539842338b6c4da69d6db6471f53552d"
format_stamp: "Formatted at 2026-09-28 06:23:13 on dist-test-slave-q03f"
server_key: "36764ec1c61570a07d22d7cbb156c30d"
server_key_iv: "f7b15102abcfc5a1266ffd6b07b738e7"
server_key_version: "encryptionkey@0"
I20260928 06:23:13.876331 4413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "539842338b6c4da69d6db6471f53552d"
format_stamp: "Formatted at 2026-09-28 06:23:13 on dist-test-slave-q03f"
server_key: "36764ec1c61570a07d22d7cbb156c30d"
server_key_iv: "f7b15102abcfc5a1266ffd6b07b738e7"
server_key_version: "encryptionkey@0"
I20260928 06:23:13.877849 4413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 06:23:13.878818 4427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:13.879113 4413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:13.879242 4413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "539842338b6c4da69d6db6471f53552d"
format_stamp: "Formatted at 2026-09-28 06:23:13 on dist-test-slave-q03f"
server_key: "36764ec1c61570a07d22d7cbb156c30d"
server_key_iv: "f7b15102abcfc5a1266ffd6b07b738e7"
server_key_version: "encryptionkey@0"
I20260928 06:23:13.879382 4413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:13.889673 4413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:13.892047 4413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:13.892239 4413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:13.892534 4413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:13.892890 4413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:13.892948 4413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:13.892985 4413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:13.893013 4413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:13.899272 4413 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:38499
I20260928 06:23:13.899317 4540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:38499 every 8 connection(s)
I20260928 06:23:13.899600 4413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:23:13.905537 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 4413
I20260928 06:23:13.905685 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:13.905901 32601 external_mini_cluster.cc:1428] Setting key 1c5c64ebec3f5a8a5708fde19b7ce927
Sep 28 06:23:13 dist-test-slave-q03f krb5kdc[4174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576593, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:13.910897 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:13.900399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:58717 (local address 127.31.214.126:37641)
0928 06:23:13.900478 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:13.900480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:13.900776 (+ 296us) server_negotiation.cc:407] Connection header received
0928 06:23:13.901318 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:13.901319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:13.901331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:13.901415 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:13.902529 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.903068 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.904157 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.904506 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.904582 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:13.907312 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:13.907314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:13.907314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:13.907322 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:13.909176 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:13.909699 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:13.909701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:13.909701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:13.909767 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:13.910106 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:13.910107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:13.910107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:13.910245 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:13.910365 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:13.910754 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 06:23:13.910828 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:23:13.911188 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:13.900416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37641 (local address 127.31.214.66:58717)
0928 06:23:13.900657 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:13.900695 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:13.901220 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:13.901490 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:13.901494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:13.901754 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:13.902367 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:13.902373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.903219 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.903220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:13.904079 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:13.904081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:13.904589 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:13.904590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:13.904734 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:13.905336 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:13.905345 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:13.907212 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:13.909278 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:13.909285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:13.909288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:13.909613 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:13.909843 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:13.909844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:13.909844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:13.910024 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:13.910365 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:13.910373 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:13.910571 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 06:23:13.910697 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 06:23:13.910888 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
I20260928 06:23:13.911724 4541 heartbeater.cc:347] Connected to a master server at 127.31.214.126:37641
I20260928 06:23:13.911873 4541 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:13.912130 4541 heartbeater.cc:511] Master 127.31.214.126:37641 requested a full tablet report, sending...
I20260928 06:23:13.912629 4218 ts_manager.cc:194] Registered new tserver with Master: 539842338b6c4da69d6db6471f53552d (127.31.214.66:38499)
I20260928 06:23:13.913090 4218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:58717
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:23:13.933506 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:37641
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:14.021359 4546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:14.021602 4546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:14.021628 4546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:14.022971 4546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:14.023026 4546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:14.023056 4546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:14.025215 4546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:37641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:14.025558 4546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:14.025813 4546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:14.031608 4554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.031639 4551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.031629 4552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:14.031864 4546 server_base.cc:1034] running on GCE node
I20260928 06:23:14.032459 4546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:14.032822 4546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:14.034037 4546 hybrid_clock.cc:648] HybridClock initialized: now 1790576594034029 us; error 40 us; skew 500 ppm
Sep 28 06:23:14 dist-test-slave-q03f krb5kdc[4174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576594, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:14.036876 4546 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:23:14.037276 4546 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:14.037349 4546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:14.037567 4546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:14.038777 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "9b4e290a56584cf384569b877cb1d345"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "e09bd69496e426a92cc5df5cefffe5ad"
server_key_iv: "7c223c27fc997de174ba87b80b77a2c7"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.039172 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "9b4e290a56584cf384569b877cb1d345"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "e09bd69496e426a92cc5df5cefffe5ad"
server_key_iv: "7c223c27fc997de174ba87b80b77a2c7"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.040580 4546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:23:14.041486 4560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.041643 4546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:14.041754 4546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "9b4e290a56584cf384569b877cb1d345"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "e09bd69496e426a92cc5df5cefffe5ad"
server_key_iv: "7c223c27fc997de174ba87b80b77a2c7"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.041833 4546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:14.056834 4546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:14.059500 4546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:14.059664 4546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:14.059955 4546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:14.060415 4546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:14.060475 4546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.060532 4546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:14.060559 4546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.067077 4546 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:40801
I20260928 06:23:14.067109 4673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:40801 every 8 connection(s)
I20260928 06:23:14.067427 4546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:14.069352 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 4546
I20260928 06:23:14.069515 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:14.069732 32601 external_mini_cluster.cc:1428] Setting key cab1fcbebcce0c8306eff576c5d5cf87
Sep 28 06:23:14 dist-test-slave-q03f krb5kdc[4174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576594, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:14.077839 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.068218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:53369 (local address 127.31.214.126:37641)
0928 06:23:14.068291 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.068294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.068529 (+ 235us) server_negotiation.cc:407] Connection header received
0928 06:23:14.069037 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.069038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.069049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.069117 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.070578 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.071148 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.072155 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.072350 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.072443 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.074906 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.074908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.074908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:14.074916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.076454 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:14.076885 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:14.076885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:14.076886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:14.076944 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:14.077220 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:14.077221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:14.077221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:14.077338 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.077416 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.077684 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.077755 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:23:14.078089 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.068186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37641 (local address 127.31.214.67:53369)
0928 06:23:14.068385 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.068431 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.068938 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.069252 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.069258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.069679 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:14.070469 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.070476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.071233 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.071234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.072074 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.072077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.072486 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.072488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.072621 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.073219 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:14.073228 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:14.074810 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:14.076551 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:14.076558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:14.076560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:14.076799 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:14.077029 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:14.077030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:14.077031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:14.077151 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:14.077426 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:14.077430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:14.077560 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.077654 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.077795 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260928 06:23:14.078703 4674 heartbeater.cc:347] Connected to a master server at 127.31.214.126:37641
I20260928 06:23:14.078830 4674 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:14.079033 4674 heartbeater.cc:511] Master 127.31.214.126:37641 requested a full tablet report, sending...
I20260928 06:23:14.079483 4218 ts_manager.cc:194] Registered new tserver with Master: 9b4e290a56584cf384569b877cb1d345 (127.31.214.67:40801)
I20260928 06:23:14.079969 4218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:53369
I20260928 06:23:14.081555 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:23:14.089329 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.083076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.31.214.126:37641)
0928 06:23:14.083158 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.083161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.083307 (+ 146us) server_negotiation.cc:407] Connection header received
0928 06:23:14.083333 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.083334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.083380 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.083448 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.084051 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.084570 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.085308 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.085513 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.085570 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.086919 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.086921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.086921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:14.086928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.088264 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:14.088611 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:14.088612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:14.088612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:14.088663 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:14.088833 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:14.088833 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:14.088834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:14.088998 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.089077 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.089166 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.089233 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:23:14.091243 4218 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:14.092998 4218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:14.100353 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.096057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.31.214.65:42337)
0928 06:23:14.096333 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.096337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.096390 (+ 53us) server_negotiation.cc:407] Connection header received
0928 06:23:14.096635 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.096638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.096655 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.096734 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:14.096834 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.097538 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.099184 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.099881 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.099957 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.100004 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.100211 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.100255 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260928 06:23:14.100360 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.095918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:42337 (local address 127.0.0.1:58544)
0928 06:23:14.096330 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.096339 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.096402 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.096730 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.096731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.096733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:14.096779 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.096781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.097648 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.097650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.099093 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.099094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.099970 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.099971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.100075 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.100095 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.100182 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.100254 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":39,"thread_start_us":66,"threads_started":1}
I20260928 06:23:14.100364 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.095718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:40801 (local address 127.0.0.1:32930)
0928 06:23:14.096238 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.096249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.096414 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.096744 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.096745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.096747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:14.096793 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.096795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.098029 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.098030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.098979 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.098981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.099967 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.099968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.100075 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.100095 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.100182 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.100254 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":6,"thread_start_us":92,"threads_started":1}
I20260928 06:23:14.100680 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.096060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.31.214.67:40801)
0928 06:23:14.096242 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.096247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.096515 (+ 268us) server_negotiation.cc:407] Connection header received
0928 06:23:14.096562 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.096565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.096581 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.096657 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:14.096871 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.097924 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.099081 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.099830 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.099915 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.099987 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.100523 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.100584 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260928 06:23:14.101290 4342 tablet_service.cc:1467] Processing CreateTablet for tablet df974d35484c456087564a01bc7b2a5b (DEFAULT_TABLE table=test-table [id=5835236a2ddd4849876b930919989547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:14.101361 4608 tablet_service.cc:1467] Processing CreateTablet for tablet df974d35484c456087564a01bc7b2a5b (DEFAULT_TABLE table=test-table [id=5835236a2ddd4849876b930919989547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:14.101617 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.095718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:38499 (local address 127.0.0.1:53224)
0928 06:23:14.095875 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.095924 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.095999 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.096383 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.096387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.096390 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:14.096469 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.096471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.097919 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.097920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.099092 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.099094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.101306 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.101307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.101423 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.101427 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.101496 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.101545 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":27,"thread_start_us":54,"threads_started":1}
I20260928 06:23:14.101716 4342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df974d35484c456087564a01bc7b2a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:14.101737 4608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df974d35484c456087564a01bc7b2a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:14.101977 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.095854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.31.214.66:38499)
0928 06:23:14.096079 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.096084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.096094 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:23:14.096130 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.096132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.096150 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.096230 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:14.096541 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.097799 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.100041 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.101214 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.101296 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.101355 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.101648 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.101875 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
I20260928 06:23:14.102846 4475 tablet_service.cc:1467] Processing CreateTablet for tablet df974d35484c456087564a01bc7b2a5b (DEFAULT_TABLE table=test-table [id=5835236a2ddd4849876b930919989547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:14.103190 4475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df974d35484c456087564a01bc7b2a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:14.104904 4691 tablet_bootstrap.cc:492] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce: Bootstrap starting.
I20260928 06:23:14.105438 4692 tablet_bootstrap.cc:492] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: Bootstrap starting.
I20260928 06:23:14.105860 4691 tablet_bootstrap.cc:654] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:14.106155 4692 tablet_bootstrap.cc:654] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:14.106570 4693 tablet_bootstrap.cc:492] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: Bootstrap starting.
I20260928 06:23:14.106586 4691 log.cc:824] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:14.106961 4692 log.cc:824] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:14.107323 4693 tablet_bootstrap.cc:654] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:14.107689 4691 tablet_bootstrap.cc:492] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce: No bootstrap required, opened a new log
I20260928 06:23:14.107748 4692 tablet_bootstrap.cc:492] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: No bootstrap required, opened a new log
I20260928 06:23:14.107772 4691 ts_tablet_manager.cc:1397] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 06:23:14.107798 4692 ts_tablet_manager.cc:1397] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:23:14.107932 4693 log.cc:824] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:14.108650 4693 tablet_bootstrap.cc:492] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: No bootstrap required, opened a new log
I20260928 06:23:14.108735 4693 ts_tablet_manager.cc:1397] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:23:14.109793 4692 raft_consensus.cc:348] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } }
I20260928 06:23:14.109793 4691 raft_consensus.cc:348] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } }
I20260928 06:23:14.109995 4691 raft_consensus.cc:374] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:14.109995 4692 raft_consensus.cc:374] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:14.110041 4692 raft_consensus.cc:729] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4e290a56584cf384569b877cb1d345, State: Initialized, Role: FOLLOWER
I20260928 06:23:14.110041 4691 raft_consensus.cc:729] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8804fa75084ed297c97825bf2537ce, State: Initialized, Role: FOLLOWER
I20260928 06:23:14.110167 4692 consensus_queue.cc:260] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } }
I20260928 06:23:14.110162 4691 consensus_queue.cc:260] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } }
I20260928 06:23:14.110456 4692 ts_tablet_manager.cc:1428] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:23:14.110456 4691 ts_tablet_manager.cc:1428] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:23:14.110610 4408 heartbeater.cc:503] Master 127.31.214.126:37641 was elected leader, sending a full tablet report...
I20260928 06:23:14.110608 4674 heartbeater.cc:503] Master 127.31.214.126:37641 was elected leader, sending a full tablet report...
I20260928 06:23:14.110441 4693 raft_consensus.cc:348] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } }
I20260928 06:23:14.110932 4693 raft_consensus.cc:374] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:14.111007 4693 raft_consensus.cc:729] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539842338b6c4da69d6db6471f53552d, State: Initialized, Role: FOLLOWER
I20260928 06:23:14.111129 4693 consensus_queue.cc:260] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } }
I20260928 06:23:14.111459 4541 heartbeater.cc:503] Master 127.31.214.126:37641 was elected leader, sending a full tablet report...
I20260928 06:23:14.111687 4693 ts_tablet_manager.cc:1428] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:23:14.124691 4698 raft_consensus.cc:482] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:14.124812 4698 raft_consensus.cc:504] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } }
I20260928 06:23:14.125239 4698 leader_election.cc:288] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 539842338b6c4da69d6db6471f53552d (127.31.214.66:38499), 9b4e290a56584cf384569b877cb1d345 (127.31.214.67:40801)
I20260928 06:23:14.129349 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.125584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:57803 (local address 127.31.214.67:40801)
0928 06:23:14.125624 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.125627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.125694 (+ 67us) server_negotiation.cc:407] Connection header received
0928 06:23:14.125759 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.125760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.125768 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.125822 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:14.126017 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.126746 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.128271 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.128913 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.129004 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.129014 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.129233 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.129274 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:23:14.129349 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.125505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:47243 (local address 127.31.214.66:38499)
0928 06:23:14.125554 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.125556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.125567 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:23:14.125628 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.125629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.125640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.125711 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:14.125915 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.126747 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.128271 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.128913 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.129008 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.129017 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.129233 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.129274 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:23:14.129372 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.125426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:38499 (local address 127.31.214.65:47243)
0928 06:23:14.125503 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.125517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.125559 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.125724 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.125725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.125727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:14.125825 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.125827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.126844 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.126846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.128165 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.128168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.129039 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.129041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.129136 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.129144 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.129214 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.129266 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":72}
I20260928 06:23:14.129379 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.125446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:40801 (local address 127.31.214.65:57803)
0928 06:23:14.125624 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.125633 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.125706 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.125843 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.125844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.125846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:14.125912 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.125914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.126844 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.126846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.128165 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:14.128168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.129039 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.129041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:14.129136 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.129144 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.129214 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.129268 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":18,"thread_start_us":118,"threads_started":1}
I20260928 06:23:14.129827 4495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df974d35484c456087564a01bc7b2a5b" candidate_uuid: "7c8804fa75084ed297c97825bf2537ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539842338b6c4da69d6db6471f53552d" is_pre_election: true
I20260928 06:23:14.129866 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df974d35484c456087564a01bc7b2a5b" candidate_uuid: "7c8804fa75084ed297c97825bf2537ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4e290a56584cf384569b877cb1d345" is_pre_election: true
I20260928 06:23:14.130031 4628 raft_consensus.cc:2454] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8804fa75084ed297c97825bf2537ce in term 0.
I20260928 06:23:14.130031 4495 raft_consensus.cc:2454] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8804fa75084ed297c97825bf2537ce in term 0.
I20260928 06:23:14.130295 4295 leader_election.cc:302] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 539842338b6c4da69d6db6471f53552d, 7c8804fa75084ed297c97825bf2537ce; no voters:
I20260928 06:23:14.130509 4698 raft_consensus.cc:2781] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:14.130553 4698 raft_consensus.cc:482] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:14.130596 4698 raft_consensus.cc:3037] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:14.131456 4698 raft_consensus.cc:504] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } }
I20260928 06:23:14.131634 4698 leader_election.cc:288] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [CANDIDATE]: Term 1 election: Requested vote from peers 539842338b6c4da69d6db6471f53552d (127.31.214.66:38499), 9b4e290a56584cf384569b877cb1d345 (127.31.214.67:40801)
I20260928 06:23:14.131798 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df974d35484c456087564a01bc7b2a5b" candidate_uuid: "7c8804fa75084ed297c97825bf2537ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b4e290a56584cf384569b877cb1d345"
I20260928 06:23:14.131794 4495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df974d35484c456087564a01bc7b2a5b" candidate_uuid: "7c8804fa75084ed297c97825bf2537ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539842338b6c4da69d6db6471f53552d"
I20260928 06:23:14.131873 4628 raft_consensus.cc:3037] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:14.131873 4495 raft_consensus.cc:3037] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:14.132812 4628 raft_consensus.cc:2454] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8804fa75084ed297c97825bf2537ce in term 1.
I20260928 06:23:14.132851 4495 raft_consensus.cc:2454] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8804fa75084ed297c97825bf2537ce in term 1.
I20260928 06:23:14.132984 4296 leader_election.cc:302] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8804fa75084ed297c97825bf2537ce, 9b4e290a56584cf384569b877cb1d345; no voters:
I20260928 06:23:14.133093 4698 raft_consensus.cc:2781] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:14.133306 4698 raft_consensus.cc:686] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [term 1 LEADER]: Becoming Leader. State: Replica: 7c8804fa75084ed297c97825bf2537ce, State: Running, Role: LEADER
I20260928 06:23:14.133409 4698 consensus_queue.cc:237] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } }
I20260928 06:23:14.134346 4216 catalog_manager.cc:5495] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c8804fa75084ed297c97825bf2537ce (127.31.214.65). New cstate: current_term: 1 leader_uuid: "7c8804fa75084ed297c97825bf2537ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8804fa75084ed297c97825bf2537ce" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 42337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:14.146416 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.143932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58558 (local address 127.31.214.65:42337)
0928 06:23:14.144001 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.144003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.144019 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:23:14.144053 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.144054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.144071 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.144136 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:14.144291 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.144776 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.145677 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:14.145863 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:14.145916 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:14.146110 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:14.146215 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:14.146279 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.146311 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.146330 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:23:14.148407 4628 raft_consensus.cc:1261] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [term 1 FOLLOWER]: Refusing update from remote peer 7c8804fa75084ed297c97825bf2537ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:14.148407 4495 raft_consensus.cc:1261] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [term 1 FOLLOWER]: Refusing update from remote peer 7c8804fa75084ed297c97825bf2537ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:14.148803 4698 consensus_queue.cc:1035] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b4e290a56584cf384569b877cb1d345" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:14.149039 4701 consensus_queue.cc:1035] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "539842338b6c4da69d6db6471f53552d" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:23:14.150082 4542 tablet.cc:2369] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:23:14.153491 4706 mvcc.cc:205] Tried to move back new op lower bound from 7334201729628393472 to 7334201729570836480. Current Snapshot: MvccSnapshot[applied={T|T < 7334201729628393472}]
I20260928 06:23:14.153803 4704 mvcc.cc:205] Tried to move back new op lower bound from 7334201729628393472 to 7334201729570836480. Current Snapshot: MvccSnapshot[applied={T|T < 7334201729628393472 or (T in {7334201729628393472})}]
I20260928 06:23:14.155318 4217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:14.155431 4217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:14.156360 4217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6318b5c29cc4bf5ac209dc1baa2d541: Sending DeleteTablet for 3 replicas of tablet df974d35484c456087564a01bc7b2a5b
I20260928 06:23:14.156770 4342 tablet_service.cc:1514] Processing DeleteTablet for tablet df974d35484c456087564a01bc7b2a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:14 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:58544
I20260928 06:23:14.156824 4608 tablet_service.cc:1514] Processing DeleteTablet for tablet df974d35484c456087564a01bc7b2a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:14 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:32930
I20260928 06:23:14.156824 4475 tablet_service.cc:1514] Processing DeleteTablet for tablet df974d35484c456087564a01bc7b2a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:14 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:53224
I20260928 06:23:14.157109 4713 tablet_replica.cc:330] T df974d35484c456087564a01bc7b2a5b P 7c8804fa75084ed297c97825bf2537ce: stopping tablet replica
I20260928 06:23:14.157363 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 4280
I20260928 06:23:14.157608 4714 tablet_replica.cc:330] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: stopping tablet replica
I20260928 06:23:14.157770 4714 raft_consensus.cc:2229] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:14.158030 4714 raft_consensus.cc:2258] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:14.158411 4715 tablet_replica.cc:330] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: stopping tablet replica
I20260928 06:23:14.158532 4715 raft_consensus.cc:2229] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:14.158535 4714 ts_tablet_manager.cc:1905] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:14.158638 4715 raft_consensus.cc:2258] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:14.160298 4715 ts_tablet_manager.cc:1905] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:14.160527 4714 ts_tablet_manager.cc:1918] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:14.160626 4714 log.cc:1196] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/df974d35484c456087564a01bc7b2a5b
I20260928 06:23:14.160889 4714 ts_tablet_manager.cc:1939] T df974d35484c456087564a01bc7b2a5b P 9b4e290a56584cf384569b877cb1d345: Deleting consensus metadata
I20260928 06:23:14.161337 4203 catalog_manager.cc:4841] TS 9b4e290a56584cf384569b877cb1d345 (127.31.214.67:40801): tablet df974d35484c456087564a01bc7b2a5b (table test-table [id=5835236a2ddd4849876b930919989547]) successfully deleted
I20260928 06:23:14.162295 4715 ts_tablet_manager.cc:1918] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:14.162381 4715 log.cc:1196] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/df974d35484c456087564a01bc7b2a5b
I20260928 06:23:14.162648 4715 ts_tablet_manager.cc:1939] T df974d35484c456087564a01bc7b2a5b P 539842338b6c4da69d6db6471f53552d: Deleting consensus metadata
I20260928 06:23:14.163146 4202 catalog_manager.cc:4841] TS 539842338b6c4da69d6db6471f53552d (127.31.214.66:38499): tablet df974d35484c456087564a01bc7b2a5b (table test-table [id=5835236a2ddd4849876b930919989547]) successfully deleted
W20260928 06:23:14.164281 4202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:42337 (error 108)
I20260928 06:23:14.164603 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 4413
I20260928 06:23:14.165303 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.164767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:42337 (local address 127.0.0.1:58566)
0928 06:23:14.165201 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.165259 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:42337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404}
W20260928 06:23:14.166595 4202 catalog_manager.cc:4584] TS 7c8804fa75084ed297c97825bf2537ce (127.31.214.65:42337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df974d35484c456087564a01bc7b2a5b: Network error: Client connection negotiation failed: client connection to 127.31.214.65:42337: connect: Connection refused (error 111)
I20260928 06:23:14.170193 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 4546
I20260928 06:23:14.175019 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 4187
2026-09-28T06:23:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2059 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T06:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:14Z Disabled control of system clock
I20260928 06:23:14.203877 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38697
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:38697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:23:14.290490 4720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:14.290856 4720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:14.290889 4720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:14.292133 4720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:14.292183 4720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:14.292196 4720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:14.292213 4720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:14.292225 4720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:14.293707 4720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:38697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:14.294049 4720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:14.294448 4720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:14.300114 4725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.300127 4726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.300269 4728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:14.300993 4720 server_base.cc:1034] running on GCE node
I20260928 06:23:14.301301 4720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:14.301704 4720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:14.302879 4720 hybrid_clock.cc:648] HybridClock initialized: now 1790576594302850 us; error 42 us; skew 500 ppm
I20260928 06:23:14.304610 4720 webserver.cc:460] Webserver started at http://127.31.214.126:44587/ using document root <none> and password file <none>
I20260928 06:23:14.304991 4720 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:14.305079 4720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:14.305243 4720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:14.306507 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "0ae0ec88d68d4de0b31fa4865cd38154"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "ae8b7f9c4836b349985cbfffedf3b95e"
server_key_iv: "9870a8446e04cc66dd0b573722fa543e"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.306902 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "0ae0ec88d68d4de0b31fa4865cd38154"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "ae8b7f9c4836b349985cbfffedf3b95e"
server_key_iv: "9870a8446e04cc66dd0b573722fa543e"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.309091 4720 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:23:14.310065 4734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.310257 4720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:14.310348 4720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "0ae0ec88d68d4de0b31fa4865cd38154"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "ae8b7f9c4836b349985cbfffedf3b95e"
server_key_iv: "9870a8446e04cc66dd0b573722fa543e"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.310441 4720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:14.315982 4720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:14.316344 4720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:14.316498 4720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:14.321534 4720 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:38697
I20260928 06:23:14.321559 4786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:38697 every 8 connection(s)
I20260928 06:23:14.322078 4720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:14.323045 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:14.327324 4787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154: Bootstrap starting.
I20260928 06:23:14.328167 4787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:14.328477 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 4720
I20260928 06:23:14.328543 4787 log.cc:824] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:14.328591 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:14.328811 32601 external_mini_cluster.cc:1428] Setting key 84a155b6621c9963b27695d5c7d99374
I20260928 06:23:14.329388 4787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154: No bootstrap required, opened a new log
I20260928 06:23:14.332561 4787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae0ec88d68d4de0b31fa4865cd38154" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38697 } }
I20260928 06:23:14.332751 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.329399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.31.214.126:38697)
0928 06:23:14.329616 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.329625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.329661 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:23:14.331200 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.331215 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.331283 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.331442 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.332140 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.332144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.332144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.332148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.332201 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.332213 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.332289 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.332362 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.332429 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260928 06:23:14.332815 4787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:14.332871 4787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae0ec88d68d4de0b31fa4865cd38154, State: Initialized, Role: FOLLOWER
I20260928 06:23:14.332996 4787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae0ec88d68d4de0b31fa4865cd38154" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38697 } }
I20260928 06:23:14.333093 4787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:14.333140 4787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:14.333199 4787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:14.334676 4787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae0ec88d68d4de0b31fa4865cd38154" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38697 } }
I20260928 06:23:14.334815 4787 leader_election.cc:302] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ae0ec88d68d4de0b31fa4865cd38154; no voters:
I20260928 06:23:14.335034 4787 leader_election.cc:288] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:14.335091 4792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:14.335243 4792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [term 1 LEADER]: Becoming Leader. State: Replica: 0ae0ec88d68d4de0b31fa4865cd38154, State: Running, Role: LEADER
I20260928 06:23:14.335333 4787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:14.335363 4792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae0ec88d68d4de0b31fa4865cd38154" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38697 } }
I20260928 06:23:14.335740 4793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ae0ec88d68d4de0b31fa4865cd38154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae0ec88d68d4de0b31fa4865cd38154" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38697 } } }
I20260928 06:23:14.335841 4793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:14.336141 4794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ae0ec88d68d4de0b31fa4865cd38154. Latest consensus state: current_term: 1 leader_uuid: "0ae0ec88d68d4de0b31fa4865cd38154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ae0ec88d68d4de0b31fa4865cd38154" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38697 } } }
I20260928 06:23:14.336223 4794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:14.336347 4797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:14.337189 4797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:14.338887 4797 catalog_manager.cc:1347] Generated new cluster ID: 2aa2375448e5491a85dde73202e56516
I20260928 06:23:14.338980 4797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:14.358421 4797 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:14.358644 4797 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:14.366641 4797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154: Generated new TSK 0
I20260928 06:23:14.366901 4797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:23:14.374235 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38697
--builtin_ntp_servers=127.31.214.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:23:14.461270 4811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:14.461527 4811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:14.461555 4811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:14.462957 4811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:14.463001 4811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:14.463030 4811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:14.464449 4811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:14.464707 4811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:14.465036 4811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:14.471096 4817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.471109 4819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.471457 4816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:14.471808 4811 server_base.cc:1034] running on GCE node
I20260928 06:23:14.472018 4811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:14.472258 4811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:14.473513 4811 hybrid_clock.cc:648] HybridClock initialized: now 1790576594473504 us; error 32 us; skew 500 ppm
I20260928 06:23:14.475035 4811 webserver.cc:460] Webserver started at http://127.31.214.65:32769/ using document root <none> and password file <none>
I20260928 06:23:14.475327 4811 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:14.475400 4811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:14.475553 4811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:14.476696 4811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "6b86465461a54cabbbc7c0fc9ed29829"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "9295258b76ccd65812be7247f71b1bbf"
server_key_iv: "cffcf7892569229b61fef7b0e486fc6e"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.477085 4811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "6b86465461a54cabbbc7c0fc9ed29829"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "9295258b76ccd65812be7247f71b1bbf"
server_key_iv: "cffcf7892569229b61fef7b0e486fc6e"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.478672 4811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:23:14.479590 4825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.479815 4811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:14.479924 4811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "6b86465461a54cabbbc7c0fc9ed29829"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "9295258b76ccd65812be7247f71b1bbf"
server_key_iv: "cffcf7892569229b61fef7b0e486fc6e"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.480006 4811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:14.483769 4811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:14.484086 4811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:14.484236 4811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:14.484470 4811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:14.484808 4811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:14.484861 4811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.484905 4811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:14.484937 4811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.491180 4811 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:46817
I20260928 06:23:14.491223 4938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:46817 every 8 connection(s)
I20260928 06:23:14.491619 4811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:23:14.495144 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.492632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:35509 (local address 127.31.214.126:38697)
0928 06:23:14.492709 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.492712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.492994 (+ 282us) server_negotiation.cc:407] Connection header received
0928 06:23:14.493563 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.493565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.493570 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.493629 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.494936 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.494937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.494937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.494942 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.494956 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.494960 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.495008 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.495072 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.495086 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:23:14.495368 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.492616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38697 (local address 127.31.214.65:35509)
0928 06:23:14.492852 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.492896 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.493465 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.493688 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.493692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.493701 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:14.494770 (+ 1069us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:23:14.494771 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:14.494804 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:23:14.494837 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:14.495008 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:14.495010 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:14.495013 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.495071 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.495126 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":111,"threads_started":1}
I20260928 06:23:14.495935 4939 heartbeater.cc:347] Connected to a master server at 127.31.214.126:38697
I20260928 06:23:14.496048 4939 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:14.496241 4939 heartbeater.cc:511] Master 127.31.214.126:38697 requested a full tablet report, sending...
I20260928 06:23:14.496841 4751 ts_manager.cc:194] Registered new tserver with Master: 6b86465461a54cabbbc7c0fc9ed29829 (127.31.214.65:46817)
I20260928 06:23:14.499536 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 4811
I20260928 06:23:14.499665 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:23:14.499890 32601 external_mini_cluster.cc:1428] Setting key b8bf0fa15ce6fc723894586ddd313195
I20260928 06:23:14.500983 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38697
--builtin_ntp_servers=127.31.214.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:23:14.593907 4942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:14.594178 4942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:14.594286 4942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:14.595666 4942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:14.595718 4942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:14.595781 4942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:14.597507 4942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:14.597812 4942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:14.598205 4942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:14.603952 4947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.603937 4948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.603937 4950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:14.604337 4942 server_base.cc:1034] running on GCE node
I20260928 06:23:14.604544 4942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:14.604794 4942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:14.605993 4942 hybrid_clock.cc:648] HybridClock initialized: now 1790576594605940 us; error 36 us; skew 500 ppm
I20260928 06:23:14.607328 4942 webserver.cc:460] Webserver started at http://127.31.214.66:40643/ using document root <none> and password file <none>
I20260928 06:23:14.607591 4942 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:14.607643 4942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:14.607782 4942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:14.608920 4942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "68b42e9b721b472b9a35977e11790365"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "f10d32f70c78848af88b8d868396943b"
server_key_iv: "60a1be03a9de477be9a1d20cd6e6c3c3"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.609285 4942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "68b42e9b721b472b9a35977e11790365"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "f10d32f70c78848af88b8d868396943b"
server_key_iv: "60a1be03a9de477be9a1d20cd6e6c3c3"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.610746 4942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:14.611528 4956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.611722 4942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:14.611826 4942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "68b42e9b721b472b9a35977e11790365"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "f10d32f70c78848af88b8d868396943b"
server_key_iv: "60a1be03a9de477be9a1d20cd6e6c3c3"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.611913 4942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:14.615504 4942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:14.615792 4942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:14.615940 4942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:14.616145 4942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:14.616466 4942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:14.616519 4942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.616557 4942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:14.616585 4942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.623327 4942 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:40315
I20260928 06:23:14.623389 5069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:40315 every 8 connection(s)
I20260928 06:23:14.623739 4942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:23:14.626546 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 4942
I20260928 06:23:14.626636 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:14.626825 32601 external_mini_cluster.cc:1428] Setting key db2718dd2652aea0d2a1a7aca9bcbe11
I20260928 06:23:14.627322 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.624549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:57783 (local address 127.31.214.126:38697)
0928 06:23:14.624609 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.624611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.624818 (+ 207us) server_negotiation.cc:407] Connection header received
0928 06:23:14.625347 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.625348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.625354 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.625423 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.627037 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.627039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.627040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.627045 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.627062 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.627066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.627145 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.627242 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.627258 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:23:14.627578 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.624538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38697 (local address 127.31.214.66:57783)
0928 06:23:14.624705 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.624740 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.625235 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.625509 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.625514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.625522 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:14.626827 (+ 1305us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:23:14.626828 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:14.626882 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:23:14.626921 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:14.627150 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:14.627156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:14.627160 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.627239 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.627315 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260928 06:23:14.628058 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38697
--builtin_ntp_servers=127.31.214.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 06:23:14.628132 5070 heartbeater.cc:347] Connected to a master server at 127.31.214.126:38697
I20260928 06:23:14.628225 5070 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:14.628392 5070 heartbeater.cc:511] Master 127.31.214.126:38697 requested a full tablet report, sending...
I20260928 06:23:14.628782 4751 ts_manager.cc:194] Registered new tserver with Master: 68b42e9b721b472b9a35977e11790365 (127.31.214.66:40315)
W20260928 06:23:14.711856 5073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:14.712080 5073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:14.712103 5073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:14.713686 5073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:14.713753 5073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:14.713793 5073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:14.716215 5073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:14.716564 5073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:14.716945 5073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:14.722401 5078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.722429 5081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:14.722470 5079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:14.722610 5073 server_base.cc:1034] running on GCE node
I20260928 06:23:14.722918 5073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:14.723168 5073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:14.724313 5073 hybrid_clock.cc:648] HybridClock initialized: now 1790576594724297 us; error 32 us; skew 500 ppm
I20260928 06:23:14.725710 5073 webserver.cc:460] Webserver started at http://127.31.214.67:37729/ using document root <none> and password file <none>
I20260928 06:23:14.726039 5073 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:14.726094 5073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:14.726244 5073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:14.727334 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "f7f8208943794d6a96bb5774e9873a53"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "1d322bbd7f6e3e41413ee40af7b20d97"
server_key_iv: "72abcd78b7889cb978a4edc7dc7078f9"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.727707 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "f7f8208943794d6a96bb5774e9873a53"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "1d322bbd7f6e3e41413ee40af7b20d97"
server_key_iv: "72abcd78b7889cb978a4edc7dc7078f9"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.729084 5073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:23:14.729918 5087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.730261 5073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:14.730410 5073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "f7f8208943794d6a96bb5774e9873a53"
format_stamp: "Formatted at 2026-09-28 06:23:14 on dist-test-slave-q03f"
server_key: "1d322bbd7f6e3e41413ee40af7b20d97"
server_key_iv: "72abcd78b7889cb978a4edc7dc7078f9"
server_key_version: "encryptionkey@0"
I20260928 06:23:14.730495 5073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:14.734015 5073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:14.734292 5073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:14.734442 5073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:14.734649 5073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:14.735026 5073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:14.735080 5073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.735121 5073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:14.735146 5073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:14.741217 5073 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:46621
I20260928 06:23:14.741261 5200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:46621 every 8 connection(s)
I20260928 06:23:14.741616 5073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:14.743321 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 5073
I20260928 06:23:14.743468 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:14.743681 32601 external_mini_cluster.cc:1428] Setting key 371801975544146b6b14ce20dd9827bd
I20260928 06:23:14.745306 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.742572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:36343 (local address 127.31.214.126:38697)
0928 06:23:14.742651 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.742653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.742911 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:23:14.743491 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.743492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.743499 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.743555 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.745075 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.745076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.745076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.745080 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.745096 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.745100 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.745146 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.745239 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.745251 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:23:14.745584 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.742570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38697 (local address 127.31.214.67:36343)
0928 06:23:14.742779 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.742815 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.743361 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.743650 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.743654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.743662 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:14.744933 (+ 1271us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:23:14.744933 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:14.744964 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:23:14.744993 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:14.745185 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:14.745189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:14.745191 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.745249 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.745303 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260928 06:23:14.746182 5201 heartbeater.cc:347] Connected to a master server at 127.31.214.126:38697
I20260928 06:23:14.746281 5201 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:14.746407 5201 heartbeater.cc:511] Master 127.31.214.126:38697 requested a full tablet report, sending...
I20260928 06:23:14.746783 4751 ts_manager.cc:194] Registered new tserver with Master: f7f8208943794d6a96bb5774e9873a53 (127.31.214.67:46621)
I20260928 06:23:14.756434 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:23:14.759337 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.758158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.31.214.126:38697)
0928 06:23:14.758367 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.758369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.758380 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:23:14.758409 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.758409 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.758414 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.758477 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.759143 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.759144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.759144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.759148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.759163 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.759167 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.759225 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.759264 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.759276 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260928 06:23:14.761514 4751 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:14.763643 4751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:14.768708 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.766598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:46621 (local address 127.0.0.1:35370)
0928 06:23:14.766787 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.767294 (+ 507us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.767336 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.767608 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.767615 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.767623 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:14.768135 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:23:14.768135 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:14.768190 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:23:14.768221 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:14.768526 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:14.768533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:14.768537 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.768592 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.768608 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
I20260928 06:23:14.768795 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.766825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.31.214.66:40315)
0928 06:23:14.767178 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.767182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.767195 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:23:14.767234 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.767236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.767249 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.767322 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.768406 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.768417 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.768417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.768422 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.768457 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.768483 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.768545 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.768670 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.768691 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":108,"threads_started":1}
I20260928 06:23:14.768895 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.766852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.31.214.65:46817)
0928 06:23:14.767076 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.767094 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.767243 (+ 149us) server_negotiation.cc:407] Connection header received
0928 06:23:14.767277 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.767281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.767293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.767379 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.768512 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.768521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.768521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.768526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.768565 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.768584 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.768661 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.768783 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.768800 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":122,"threads_started":1}
I20260928 06:23:14.768708 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.766608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:40315 (local address 127.0.0.1:53454)
0928 06:23:14.767095 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.767107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.767170 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.767739 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.767740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.767741 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:14.768216 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:23:14.768216 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:14.768221 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:23:14.768230 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:14.768571 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:14.768572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:14.768573 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.768629 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.768640 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":99,"threads_started":1}
I20260928 06:23:14.769603 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.766837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.31.214.67:46621)
0928 06:23:14.767457 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.767462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.767473 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:23:14.767496 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.767497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.767505 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.767585 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.768338 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.768349 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.768349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.768355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.768404 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.768434 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.769449 (+ 1015us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.769475 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.769492 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":140,"threads_started":1}
I20260928 06:23:14.768937 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.766599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46817 (local address 127.0.0.1:51106)
0928 06:23:14.766860 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.767069 (+ 209us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.767154 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.767750 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.767751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.767754 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:14.768396 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:23:14.768397 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:14.768404 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:23:14.768418 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:14.768690 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:14.768691 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:14.768692 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.768737 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.768750 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":4,"thread_start_us":106,"threads_started":1}
I20260928 06:23:14.770277 5004 tablet_service.cc:1467] Processing CreateTablet for tablet 19f12dc0a3eb458db19e71fdd5900514 (DEFAULT_TABLE table=test-table [id=3ae924b6868d46f7b730762d4e76161f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:14.770327 5135 tablet_service.cc:1467] Processing CreateTablet for tablet 19f12dc0a3eb458db19e71fdd5900514 (DEFAULT_TABLE table=test-table [id=3ae924b6868d46f7b730762d4e76161f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:14.770586 4873 tablet_service.cc:1467] Processing CreateTablet for tablet 19f12dc0a3eb458db19e71fdd5900514 (DEFAULT_TABLE table=test-table [id=3ae924b6868d46f7b730762d4e76161f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:14.770699 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f12dc0a3eb458db19e71fdd5900514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:14.770691 5004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f12dc0a3eb458db19e71fdd5900514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:14.770907 4873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f12dc0a3eb458db19e71fdd5900514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:14.773447 5215 tablet_bootstrap.cc:492] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: Bootstrap starting.
I20260928 06:23:14.774061 5217 tablet_bootstrap.cc:492] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: Bootstrap starting.
I20260928 06:23:14.774274 5215 tablet_bootstrap.cc:654] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:14.774262 5216 tablet_bootstrap.cc:492] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829: Bootstrap starting.
I20260928 06:23:14.774680 5215 log.cc:824] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:14.774755 5217 tablet_bootstrap.cc:654] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:14.774856 5216 tablet_bootstrap.cc:654] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:14.775048 5217 log.cc:824] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:14.775166 5216 log.cc:824] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:14.775503 5215 tablet_bootstrap.cc:492] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: No bootstrap required, opened a new log
I20260928 06:23:14.775581 5215 ts_tablet_manager.cc:1397] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:14.775867 5217 tablet_bootstrap.cc:492] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: No bootstrap required, opened a new log
I20260928 06:23:14.775950 5217 ts_tablet_manager.cc:1397] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:14.777717 5215 raft_consensus.cc:348] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } }
I20260928 06:23:14.777858 5215 raft_consensus.cc:374] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:14.777881 5215 raft_consensus.cc:729] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b42e9b721b472b9a35977e11790365, State: Initialized, Role: FOLLOWER
I20260928 06:23:14.778070 5215 consensus_queue.cc:260] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } }
I20260928 06:23:14.778105 5217 raft_consensus.cc:348] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } }
I20260928 06:23:14.778333 5217 raft_consensus.cc:374] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:14.778386 5217 raft_consensus.cc:729] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f8208943794d6a96bb5774e9873a53, State: Initialized, Role: FOLLOWER
I20260928 06:23:14.778386 5215 ts_tablet_manager.cc:1428] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 06:23:14.778434 5070 heartbeater.cc:503] Master 127.31.214.126:38697 was elected leader, sending a full tablet report...
I20260928 06:23:14.778510 5217 consensus_queue.cc:260] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } }
I20260928 06:23:14.778779 5217 ts_tablet_manager.cc:1428] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:23:14.778790 5201 heartbeater.cc:503] Master 127.31.214.126:38697 was elected leader, sending a full tablet report...
I20260928 06:23:14.779505 5216 tablet_bootstrap.cc:492] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829: No bootstrap required, opened a new log
I20260928 06:23:14.779580 5216 ts_tablet_manager.cc:1397] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:23:14.781788 5216 raft_consensus.cc:348] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } }
I20260928 06:23:14.781929 5216 raft_consensus.cc:374] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:14.782016 5216 raft_consensus.cc:729] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b86465461a54cabbbc7c0fc9ed29829, State: Initialized, Role: FOLLOWER
I20260928 06:23:14.782120 5216 consensus_queue.cc:260] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } }
I20260928 06:23:14.782377 5216 ts_tablet_manager.cc:1428] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:23:14.782500 4939 heartbeater.cc:503] Master 127.31.214.126:38697 was elected leader, sending a full tablet report...
W20260928 06:23:14.874380 5071 tablet.cc:2369] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:23:14.905535 5223 raft_consensus.cc:482] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:14.905678 5223 raft_consensus.cc:504] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } }
I20260928 06:23:14.906277 5223 leader_election.cc:288] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7f8208943794d6a96bb5774e9873a53 (127.31.214.67:46621), 68b42e9b721b472b9a35977e11790365 (127.31.214.66:40315)
I20260928 06:23:14.907910 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.906571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:40315 (local address 127.31.214.65:52547)
0928 06:23:14.906647 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.906661 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.906727 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.906928 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.906929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.906932 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:14.907484 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:23:14.907485 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:14.907493 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:23:14.907508 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:14.907732 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:14.907733 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:14.907734 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.907812 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.907837 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:23:14.907910 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.906620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:52547 (local address 127.31.214.66:40315)
0928 06:23:14.906670 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.906672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.906712 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:23:14.906827 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.906829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.906836 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.906921 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.907606 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.907608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.907608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.907612 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.907652 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.907657 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.907724 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.907817 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.907833 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:23:14.908054 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.906650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:59297 (local address 127.31.214.67:46621)
0928 06:23:14.906711 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.906713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.906899 (+ 186us) server_negotiation.cc:407] Connection header received
0928 06:23:14.906965 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.906966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.906971 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.907046 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.907797 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.907798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.907798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.907803 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.907818 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.907822 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.907895 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.907943 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.907959 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:23:14.908182 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.906579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:46621 (local address 127.31.214.65:59297)
0928 06:23:14.906812 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.906826 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:14.906902 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:14.907049 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:14.907050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:14.907053 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:14.907703 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:23:14.907703 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:14.907710 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:23:14.907723 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:14.907892 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:14.907893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:14.907893 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 06:23:14.907946 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:23:14.908081 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":64,"thread_start_us":112,"threads_started":1}
I20260928 06:23:14.908608 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f12dc0a3eb458db19e71fdd5900514" candidate_uuid: "6b86465461a54cabbbc7c0fc9ed29829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f8208943794d6a96bb5774e9873a53" is_pre_election: true
I20260928 06:23:14.908581 5024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f12dc0a3eb458db19e71fdd5900514" candidate_uuid: "6b86465461a54cabbbc7c0fc9ed29829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b42e9b721b472b9a35977e11790365" is_pre_election: true
I20260928 06:23:14.908764 5155 raft_consensus.cc:2454] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b86465461a54cabbbc7c0fc9ed29829 in term 0.
I20260928 06:23:14.908790 5024 raft_consensus.cc:2454] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b86465461a54cabbbc7c0fc9ed29829 in term 0.
I20260928 06:23:14.909018 4827 leader_election.cc:302] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b86465461a54cabbbc7c0fc9ed29829, f7f8208943794d6a96bb5774e9873a53; no voters:
I20260928 06:23:14.909157 5223 raft_consensus.cc:2781] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:14.909214 5223 raft_consensus.cc:482] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:14.909238 5223 raft_consensus.cc:3037] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:14.910069 5223 raft_consensus.cc:504] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } }
I20260928 06:23:14.910223 5223 leader_election.cc:288] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [CANDIDATE]: Term 1 election: Requested vote from peers f7f8208943794d6a96bb5774e9873a53 (127.31.214.67:46621), 68b42e9b721b472b9a35977e11790365 (127.31.214.66:40315)
I20260928 06:23:14.910411 5024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f12dc0a3eb458db19e71fdd5900514" candidate_uuid: "6b86465461a54cabbbc7c0fc9ed29829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68b42e9b721b472b9a35977e11790365"
I20260928 06:23:14.910482 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f12dc0a3eb458db19e71fdd5900514" candidate_uuid: "6b86465461a54cabbbc7c0fc9ed29829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f8208943794d6a96bb5774e9873a53"
I20260928 06:23:14.910519 5024 raft_consensus.cc:3037] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:14.910557 5155 raft_consensus.cc:3037] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:14.911448 5024 raft_consensus.cc:2454] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b86465461a54cabbbc7c0fc9ed29829 in term 1.
I20260928 06:23:14.911523 5155 raft_consensus.cc:2454] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b86465461a54cabbbc7c0fc9ed29829 in term 1.
I20260928 06:23:14.911684 4826 leader_election.cc:302] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68b42e9b721b472b9a35977e11790365, 6b86465461a54cabbbc7c0fc9ed29829; no voters:
I20260928 06:23:14.911801 5223 raft_consensus.cc:2781] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:14.911994 5223 raft_consensus.cc:686] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [term 1 LEADER]: Becoming Leader. State: Replica: 6b86465461a54cabbbc7c0fc9ed29829, State: Running, Role: LEADER
I20260928 06:23:14.912289 5223 consensus_queue.cc:237] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } }
I20260928 06:23:14.913372 4750 catalog_manager.cc:5495] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b86465461a54cabbbc7c0fc9ed29829 (127.31.214.65). New cstate: current_term: 1 leader_uuid: "6b86465461a54cabbbc7c0fc9ed29829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b86465461a54cabbbc7c0fc9ed29829" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:14.918309 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.917239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.31.214.65:46817)
0928 06:23:14.917281 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:14.917283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:14.917357 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:23:14.917428 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:14.917429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:14.917441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:14.917489 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:14.918099 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:14.918100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:14.918100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:14.918103 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:14.918115 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:14.918119 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:14.918170 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:14.918201 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:23:14.918212 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:23:14.920243 5155 raft_consensus.cc:1261] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [term 1 FOLLOWER]: Refusing update from remote peer 6b86465461a54cabbbc7c0fc9ed29829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:14.920384 5024 raft_consensus.cc:1261] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [term 1 FOLLOWER]: Refusing update from remote peer 6b86465461a54cabbbc7c0fc9ed29829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:14.920675 5223 consensus_queue.cc:1035] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f8208943794d6a96bb5774e9873a53" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 46621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:14.920850 5225 consensus_queue.cc:1035] T 19f12dc0a3eb458db19e71fdd5900514 P 6b86465461a54cabbbc7c0fc9ed29829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68b42e9b721b472b9a35977e11790365" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:14.923672 5229 mvcc.cc:205] Tried to move back new op lower bound from 7334201732790571008 to 7334201732761595904. Current Snapshot: MvccSnapshot[applied={T|T < 7334201732790571008}]
I20260928 06:23:14.924729 5230 mvcc.cc:205] Tried to move back new op lower bound from 7334201732790571008 to 7334201732761595904. Current Snapshot: MvccSnapshot[applied={T|T < 7334201732790571008 or (T in {7334201732790571008})}]
I20260928 06:23:14.926249 4751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:14.926358 4751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:14.927150 4751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ae0ec88d68d4de0b31fa4865cd38154: Sending DeleteTablet for 3 replicas of tablet 19f12dc0a3eb458db19e71fdd5900514
I20260928 06:23:14.927531 5135 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f12dc0a3eb458db19e71fdd5900514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:14 UTC) from {username='slave'} at 127.0.0.1:35370
I20260928 06:23:14.927618 5004 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f12dc0a3eb458db19e71fdd5900514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:14 UTC) from {username='slave'} at 127.0.0.1:53454
I20260928 06:23:14.927850 4873 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f12dc0a3eb458db19e71fdd5900514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:14 UTC) from {username='slave'} at 127.0.0.1:51106
I20260928 06:23:14.927938 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 4811
I20260928 06:23:14.929389 5238 tablet_replica.cc:330] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: stopping tablet replica
I20260928 06:23:14.929507 5238 raft_consensus.cc:2229] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:14.929628 5238 raft_consensus.cc:2258] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:14.929896 5239 tablet_replica.cc:330] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: stopping tablet replica
I20260928 06:23:14.930032 5239 raft_consensus.cc:2229] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:14.930158 5239 raft_consensus.cc:2258] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:14.931919 5238 ts_tablet_manager.cc:1905] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:14.932193 5239 ts_tablet_manager.cc:1905] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:14.933816 5239 ts_tablet_manager.cc:1918] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:14.933904 5239 log.cc:1196] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/19f12dc0a3eb458db19e71fdd5900514
I20260928 06:23:14.934015 5238 ts_tablet_manager.cc:1918] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:14.934091 5238 log.cc:1196] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/19f12dc0a3eb458db19e71fdd5900514
W20260928 06:23:14.934290 4737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:46817 (error 108)
I20260928 06:23:14.934371 5238 ts_tablet_manager.cc:1939] T 19f12dc0a3eb458db19e71fdd5900514 P f7f8208943794d6a96bb5774e9873a53: Deleting consensus metadata
I20260928 06:23:14.934437 5239 ts_tablet_manager.cc:1939] T 19f12dc0a3eb458db19e71fdd5900514 P 68b42e9b721b472b9a35977e11790365: Deleting consensus metadata
I20260928 06:23:14.934906 4736 catalog_manager.cc:4841] TS f7f8208943794d6a96bb5774e9873a53 (127.31.214.67:46621): tablet 19f12dc0a3eb458db19e71fdd5900514 (table test-table [id=3ae924b6868d46f7b730762d4e76161f]) successfully deleted
I20260928 06:23:14.934901 4735 catalog_manager.cc:4841] TS 68b42e9b721b472b9a35977e11790365 (127.31.214.66:40315): tablet 19f12dc0a3eb458db19e71fdd5900514 (table test-table [id=3ae924b6868d46f7b730762d4e76161f]) successfully deleted
I20260928 06:23:14.935142 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:14.934998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46817 (local address 127.0.0.1:51126)
0928 06:23:14.935064 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:14.935111 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:23:14.935128 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 4942
W20260928 06:23:14.935829 4737 catalog_manager.cc:4584] TS 6b86465461a54cabbbc7c0fc9ed29829 (127.31.214.65:46817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19f12dc0a3eb458db19e71fdd5900514: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46817: connect: Connection refused (error 111)
I20260928 06:23:14.940680 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 5073
I20260928 06:23:14.945365 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 4720
2026-09-28T06:23:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (769 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 06:23:14.966377 32601 mini_oidc.cc:152] Starting JWKS server
I20260928 06:23:14.967613 32601 webserver.cc:460] Webserver started at https://127.0.0.1:39339/ using document root <none> and password file <none>
I20260928 06:23:14.967722 32601 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:23:14.967906 32601 webserver.cc:460] Webserver started at http://127.0.0.1:37017/ using document root <none> and password file <none>
2026-09-28T06:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:14Z Disabled control of system clock
I20260928 06:23:14.976944 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38893
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:38893
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39339/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/testchainca.cert with env {}
W20260928 06:23:15.068897 5247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:15.069125 5247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:15.069151 5247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:15.070412 5247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:15.070451 5247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:15.070465 5247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:15.070479 5247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:15.070493 5247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:23:15.070506 5247 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:23:15.070519 5247 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39339/jwks.json
I20260928 06:23:15.072260 5247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:38893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39339/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:15.072497 5247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:15.072677 5247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:23:15.078105 5247 server_base.cc:1034] running on GCE node
W20260928 06:23:15.078075 5253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:15.078076 5252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:15.078076 5255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:15.078696 5247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:15.079013 5247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:15.080166 5247 hybrid_clock.cc:648] HybridClock initialized: now 1790576595080145 us; error 41 us; skew 500 ppm
I20260928 06:23:15.081523 5247 webserver.cc:460] Webserver started at http://127.31.214.126:45261/ using document root <none> and password file <none>
I20260928 06:23:15.081823 5247 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:15.081992 5247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:15.082161 5247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:15.083364 5247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "b22f6225b871477197c4398e62de5f95"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "a13f7495c8f70b0ebccd773f8b56f6e8"
server_key_iv: "620e37dc7d8083587eb4e8da16b4de9e"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.083783 5247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "b22f6225b871477197c4398e62de5f95"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "a13f7495c8f70b0ebccd773f8b56f6e8"
server_key_iv: "620e37dc7d8083587eb4e8da16b4de9e"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.085321 5247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:15.086429 5261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:15.086691 5247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:15.086833 5247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "b22f6225b871477197c4398e62de5f95"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "a13f7495c8f70b0ebccd773f8b56f6e8"
server_key_iv: "620e37dc7d8083587eb4e8da16b4de9e"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.086925 5247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:15.112618 5247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:15.113036 5247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:15.113222 5247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:15.117624 5247 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:38893
I20260928 06:23:15.117660 5313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:38893 every 8 connection(s)
I20260928 06:23:15.118124 5247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:15.119167 5314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:15.121598 5314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95: Bootstrap starting.
I20260928 06:23:15.122258 5314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:15.122535 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 5247
I20260928 06:23:15.122601 5314 log.cc:824] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:15.122649 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:15.122862 32601 external_mini_cluster.cc:1428] Setting key 8b155ebfe2dd212496e75d15a17cdcc2
I20260928 06:23:15.123436 5314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95: No bootstrap required, opened a new log
I20260928 06:23:15.126294 5314 raft_consensus.cc:348] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22f6225b871477197c4398e62de5f95" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38893 } }
I20260928 06:23:15.126471 5314 raft_consensus.cc:374] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:15.126502 5314 raft_consensus.cc:729] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22f6225b871477197c4398e62de5f95, State: Initialized, Role: FOLLOWER
I20260928 06:23:15.126628 5314 consensus_queue.cc:260] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22f6225b871477197c4398e62de5f95" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38893 } }
I20260928 06:23:15.126713 5314 raft_consensus.cc:388] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:15.126749 5314 raft_consensus.cc:482] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:15.126808 5314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:15.127403 5314 raft_consensus.cc:504] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22f6225b871477197c4398e62de5f95" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38893 } }
I20260928 06:23:15.127496 5314 leader_election.cc:302] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b22f6225b871477197c4398e62de5f95; no voters:
I20260928 06:23:15.127700 5314 leader_election.cc:288] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:15.127776 5319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:15.127929 5319 raft_consensus.cc:686] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [term 1 LEADER]: Becoming Leader. State: Replica: b22f6225b871477197c4398e62de5f95, State: Running, Role: LEADER
I20260928 06:23:15.128036 5314 sys_catalog.cc:564] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:15.128018 5319 consensus_queue.cc:237] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22f6225b871477197c4398e62de5f95" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38893 } }
I20260928 06:23:15.128350 5319 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b22f6225b871477197c4398e62de5f95. Latest consensus state: current_term: 1 leader_uuid: "b22f6225b871477197c4398e62de5f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22f6225b871477197c4398e62de5f95" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38893 } } }
I20260928 06:23:15.128434 5319 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:15.128358 5320 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b22f6225b871477197c4398e62de5f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22f6225b871477197c4398e62de5f95" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38893 } } }
I20260928 06:23:15.128727 5320 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:15.128665 5325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:15.129422 5325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:15.130715 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:15.123451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.31.214.126:38893)
0928 06:23:15.123688 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:15.123698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:15.123729 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:23:15.124727 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:15.124738 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:15.124821 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:15.124988 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:15.125198 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.126712 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.127818 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.128471 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.128616 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.129810 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:15.129814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:15.129815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:15.130057 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:15.130100 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:15.130116 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:15.130187 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:15.130445 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 06:23:15.130537 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260928 06:23:15.131348 5325 catalog_manager.cc:1347] Generated new cluster ID: c6b8acb0abe747ab919c900b08f7ca38
I20260928 06:23:15.131402 5325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:15.131771 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:23:15.153985 5325 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:15.154764 5325 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:15.161785 5325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b22f6225b871477197c4398e62de5f95: Generated new TSK 0
I20260928 06:23:15.162079 5325 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:23:15.200574 5344 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 06:23:15.200697 5344 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 06:23:15.201192 5343 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:15.185365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38893 (local address 127.0.0.1:53850)
0928 06:23:15.185606 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:15.185616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:15.185703 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:15.186021 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:15.186022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:15.186024 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:23:15.186098 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:15.186101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.186790 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.186791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:15.187418 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:15.187421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.187809 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.187810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:15.187873 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.187902 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:23:15.200836 (+ 12934us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39339/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:23:15.200951 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.214.126:38893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39339/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":111,"threads_started":1}
I20260928 06:23:15.201668 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:15.185439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.31.214.126:38893)
0928 06:23:15.185518 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:15.185520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:15.185688 (+ 168us) server_negotiation.cc:407] Connection header received
0928 06:23:15.185824 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:15.185825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:15.185846 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:15.186011 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:23:15.186187 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.186693 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.187516 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.187714 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.187793 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.188030 (+ 237us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:23:15.200673 (+ 12643us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39339/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:23:15.201451 (+ 778us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39339/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:23:15.201727 5318 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39339/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.214.126:38893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39339/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 06:23:15.201781 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 5247
2026-09-28T06:23:15Z chronyd exiting
I20260928 06:23:15.238870 32601 test_util.cc:175] -----------------------------------------------
I20260928 06:23:15.238971 32601 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790576570121568-32601-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (273 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 06:23:15.239725 32601 mini_oidc.cc:152] Starting JWKS server
I20260928 06:23:15.240375 32601 webserver.cc:460] Webserver started at https://127.0.0.1:46507/ using document root <none> and password file <none>
I20260928 06:23:15.240423 32601 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:23:15.240564 32601 webserver.cc:460] Webserver started at http://127.0.0.1:40603/ using document root <none> and password file <none>
2026-09-28T06:23:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:15Z Disabled control of system clock
I20260928 06:23:15.248764 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:44101
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:44101
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46507/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/testchainca.cert with env {}
W20260928 06:23:15.331523 5350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:15.331806 5350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:15.331851 5350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:15.333099 5350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:15.333178 5350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:15.333200 5350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:15.333223 5350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:15.333252 5350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:23:15.333276 5350 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:23:15.333302 5350 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46507/jwks.json
I20260928 06:23:15.334937 5350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:44101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:44101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46507/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:15.335239 5350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:15.335484 5350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:15.340873 5355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:15.340920 5356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:15.340888 5358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:15.341243 5350 server_base.cc:1034] running on GCE node
I20260928 06:23:15.341533 5350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:15.341823 5350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:15.343044 5350 hybrid_clock.cc:648] HybridClock initialized: now 1790576595343026 us; error 30 us; skew 500 ppm
I20260928 06:23:15.344614 5350 webserver.cc:460] Webserver started at http://127.31.214.126:35781/ using document root <none> and password file <none>
I20260928 06:23:15.344898 5350 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:15.344977 5350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:15.345131 5350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:15.346361 5350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "7612c46499ff40d9bde39042e6a8fddb"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "50b8418c4a7be5b0ceb44f1ec6a84a3c"
server_key_iv: "a623b5cd5d263971fbd0d7ec2654926c"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.346747 5350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "7612c46499ff40d9bde39042e6a8fddb"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "50b8418c4a7be5b0ceb44f1ec6a84a3c"
server_key_iv: "a623b5cd5d263971fbd0d7ec2654926c"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.348124 5350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:23:15.348981 5364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:15.349150 5350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:15.349261 5350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "7612c46499ff40d9bde39042e6a8fddb"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "50b8418c4a7be5b0ceb44f1ec6a84a3c"
server_key_iv: "a623b5cd5d263971fbd0d7ec2654926c"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.349411 5350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:15.374788 5350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:15.375201 5350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:15.375377 5350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:15.379683 5350 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:44101
I20260928 06:23:15.379726 5416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:44101 every 8 connection(s)
I20260928 06:23:15.380142 5350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:15.380975 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:15.383388 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 5350
I20260928 06:23:15.383489 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:15.383702 32601 external_mini_cluster.cc:1428] Setting key 7a926ba66051cf9ae49e6534ec826016
I20260928 06:23:15.384126 5417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb: Bootstrap starting.
I20260928 06:23:15.384845 5417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:15.385160 5417 log.cc:824] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:15.386067 5417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb: No bootstrap required, opened a new log
I20260928 06:23:15.388443 5417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7612c46499ff40d9bde39042e6a8fddb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44101 } }
I20260928 06:23:15.388620 5417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:15.388655 5417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7612c46499ff40d9bde39042e6a8fddb, State: Initialized, Role: FOLLOWER
I20260928 06:23:15.388787 5417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7612c46499ff40d9bde39042e6a8fddb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44101 } }
I20260928 06:23:15.388885 5417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:15.388916 5417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:15.388962 5417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:15.389775 5417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7612c46499ff40d9bde39042e6a8fddb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44101 } }
I20260928 06:23:15.389884 5417 leader_election.cc:302] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7612c46499ff40d9bde39042e6a8fddb; no voters:
I20260928 06:23:15.390091 5417 leader_election.cc:288] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:15.390156 5422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:15.390313 5422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [term 1 LEADER]: Becoming Leader. State: Replica: 7612c46499ff40d9bde39042e6a8fddb, State: Running, Role: LEADER
I20260928 06:23:15.390400 5417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:15.390427 5422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7612c46499ff40d9bde39042e6a8fddb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44101 } }
I20260928 06:23:15.390723 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:15.384362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.31.214.126:44101)
0928 06:23:15.384584 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:15.384592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:15.384619 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:23:15.385153 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:15.385161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:15.385336 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:15.385467 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:15.385779 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.387220 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.388151 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.388780 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.388908 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.389775 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:15.389779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:15.389779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:15.390044 (+ 265us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:15.390089 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:15.390105 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:15.390366 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:15.390400 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:23:15.390487 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260928 06:23:15.390843 5422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7612c46499ff40d9bde39042e6a8fddb. Latest consensus state: current_term: 1 leader_uuid: "7612c46499ff40d9bde39042e6a8fddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7612c46499ff40d9bde39042e6a8fddb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44101 } } }
I20260928 06:23:15.390909 5422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:15.391300 5423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7612c46499ff40d9bde39042e6a8fddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7612c46499ff40d9bde39042e6a8fddb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 44101 } } }
I20260928 06:23:15.391392 5423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:15.391844 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:23:15.392009 5427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:15.392916 5427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:15.394834 5427 catalog_manager.cc:1347] Generated new cluster ID: 730575a7b9484dae9bd58dac0e8d301d
I20260928 06:23:15.394901 5427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:15.405210 5427 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:15.406020 5427 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:15.411883 5427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7612c46499ff40d9bde39042e6a8fddb: Generated new TSK 0
I20260928 06:23:15.412113 5427 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:23:15.440917 5447 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 06:23:15.441027 5447 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:23:15.441344 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:15.421913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.31.214.126:44101)
0928 06:23:15.422002 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:15.422005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:15.422161 (+ 156us) server_negotiation.cc:407] Connection header received
0928 06:23:15.422244 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:15.422245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:15.422256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:15.422330 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:23:15.422511 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.423067 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.424264 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.424477 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.424537 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.424743 (+ 206us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:23:15.441079 (+ 16336us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:23:15.441251 (+ 172us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 06:23:15.441413 5420 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 06:23:15.441550 5446 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:15.421855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:44101 (local address 127.0.0.1:44614)
0928 06:23:15.422074 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:15.422088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:15.422166 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:15.422349 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:15.422350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:15.422352 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:23:15.422420 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:15.422423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.423164 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.423165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:15.424164 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:15.424167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.424565 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.424566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:15.424632 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.424639 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:23:15.441252 (+ 16613us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:23:15.441426 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.214.126:44101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":145,"threads_started":1}
I20260928 06:23:15.441746 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 5350
2026-09-28T06:23:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 06:23:15.480340 32601 mini_oidc.cc:152] Starting JWKS server
I20260928 06:23:15.481081 32601 webserver.cc:460] Webserver started at https://127.0.0.1:35387/ using document root <none> and password file <none>
I20260928 06:23:15.481156 32601 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:23:15.481348 32601 webserver.cc:460] Webserver started at http://127.0.0.1:35005/ using document root <none> and password file <none>
2026-09-28T06:23:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:15Z Disabled control of system clock
I20260928 06:23:15.489552 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45451
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:40825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:45451
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35387/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:23:15.572849 5453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:15.573118 5453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:15.573156 5453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:15.574393 5453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:15.574465 5453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:15.574487 5453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:15.574507 5453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:15.574534 5453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:23:15.574566 5453 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:23:15.574589 5453 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35387/jwks.json
I20260928 06:23:15.576028 5453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:40825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:45451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35387/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:15.576333 5453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:15.576653 5453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:15.581984 5459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:15.582108 5453 server_base.cc:1034] running on GCE node
W20260928 06:23:15.582037 5458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:15.582231 5461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:15.582581 5453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:15.582892 5453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:15.584052 5453 hybrid_clock.cc:648] HybridClock initialized: now 1790576595584027 us; error 53 us; skew 500 ppm
I20260928 06:23:15.585574 5453 webserver.cc:460] Webserver started at http://127.31.214.126:39103/ using document root <none> and password file <none>
I20260928 06:23:15.585867 5453 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:15.585917 5453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:15.586089 5453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:15.587293 5453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "afa1cec023f54903824ede2e2f3539b2"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "0e512a0f10fbfbf5bbdb09a08628e3fb"
server_key_iv: "f65af278fda9c0be5d43b0a665c1a1b7"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.587666 5453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "afa1cec023f54903824ede2e2f3539b2"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "0e512a0f10fbfbf5bbdb09a08628e3fb"
server_key_iv: "f65af278fda9c0be5d43b0a665c1a1b7"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.589090 5453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:23:15.590075 5467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:15.590262 5453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:15.590430 5453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "afa1cec023f54903824ede2e2f3539b2"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "0e512a0f10fbfbf5bbdb09a08628e3fb"
server_key_iv: "f65af278fda9c0be5d43b0a665c1a1b7"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.590517 5453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:15.613365 5453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:15.613736 5453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:15.613898 5453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:15.618114 5453 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:45451
I20260928 06:23:15.618160 5519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:45451 every 8 connection(s)
I20260928 06:23:15.618620 5453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:15.619386 5520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:15.622525 5520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2: Bootstrap starting.
I20260928 06:23:15.623238 5520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:15.623567 5520 log.cc:824] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:15.624469 5520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2: No bootstrap required, opened a new log
I20260928 06:23:15.624663 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 5453
I20260928 06:23:15.624776 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:15.624979 32601 external_mini_cluster.cc:1428] Setting key 247b00253ad1d1df91f1238aac02c9d1
I20260928 06:23:15.626914 5520 raft_consensus.cc:348] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa1cec023f54903824ede2e2f3539b2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45451 } }
I20260928 06:23:15.627122 5520 raft_consensus.cc:374] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:15.627167 5520 raft_consensus.cc:729] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa1cec023f54903824ede2e2f3539b2, State: Initialized, Role: FOLLOWER
I20260928 06:23:15.627327 5520 consensus_queue.cc:260] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa1cec023f54903824ede2e2f3539b2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45451 } }
I20260928 06:23:15.627458 5520 raft_consensus.cc:388] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:15.627509 5520 raft_consensus.cc:482] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:15.627566 5520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:15.628218 5520 raft_consensus.cc:504] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa1cec023f54903824ede2e2f3539b2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45451 } }
I20260928 06:23:15.628351 5520 leader_election.cc:302] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afa1cec023f54903824ede2e2f3539b2; no voters:
I20260928 06:23:15.628584 5520 leader_election.cc:288] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:15.628615 5525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:15.628736 5525 raft_consensus.cc:686] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [term 1 LEADER]: Becoming Leader. State: Replica: afa1cec023f54903824ede2e2f3539b2, State: Running, Role: LEADER
I20260928 06:23:15.628830 5525 consensus_queue.cc:237] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa1cec023f54903824ede2e2f3539b2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45451 } }
I20260928 06:23:15.628912 5520 sys_catalog.cc:564] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:15.629200 5527 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afa1cec023f54903824ede2e2f3539b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa1cec023f54903824ede2e2f3539b2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45451 } } }
I20260928 06:23:15.629298 5527 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:15.629459 5526 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afa1cec023f54903824ede2e2f3539b2. Latest consensus state: current_term: 1 leader_uuid: "afa1cec023f54903824ede2e2f3539b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa1cec023f54903824ede2e2f3539b2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45451 } } }
I20260928 06:23:15.629522 5526 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:15.629631 5529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:15.630388 5529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:15.632547 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:15.625573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.31.214.126:45451)
0928 06:23:15.625780 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:15.625788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:15.625825 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:23:15.626890 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:15.626897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:15.626961 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:15.627080 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:15.627435 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.628904 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.630068 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.630671 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.631156 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.631672 (+ 516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:15.631676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:15.631676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:15.631909 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:15.631958 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:15.631975 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:15.632054 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:15.632162 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:23:15.632283 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260928 06:23:15.632563 5529 catalog_manager.cc:1347] Generated new cluster ID: ff82be42cb10491fa4fe5993bb901d81
I20260928 06:23:15.632628 5529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:15.633744 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:23:15.636567 5529 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:15.637398 5529 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:15.643119 5529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afa1cec023f54903824ede2e2f3539b2: Generated new TSK 0
I20260928 06:23:15.643333 5529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:23:15.657294 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:15.653140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.31.214.126:45451)
0928 06:23:15.653206 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:15.653209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:15.653278 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:23:15.653370 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:15.653371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:15.653384 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:15.653457 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:23:15.653586 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.654404 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.655439 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.655769 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.655884 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.656005 (+ 121us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:23:15.657028 (+ 1023us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35387/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:23:15.657165 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35387/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 06:23:15.657366 5524 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35387/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 06:23:15.659291 5549 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:15.653043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45451 (local address 127.0.0.1:48428)
0928 06:23:15.653215 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:15.653226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:15.653293 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:15.653463 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:15.653463 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:15.653465 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:23:15.653519 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:15.653522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.654742 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.654743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:15.655353 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:15.655355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.655853 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.655854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:15.655896 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.655917 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:23:15.657136 (+ 1219us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35387/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:23:15.659193 (+ 2057us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.214.126:45451: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35387/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":1832,"thread_start_us":115,"threads_started":1}
I20260928 06:23:15.659525 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 5453
E20260928 06:23:15.671833 5550 webserver.cc:533] Webserver: sslize failed:
E20260928 06:23:15.671919 5550 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T06:23:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T06:23:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:15Z Disabled control of system clock
I20260928 06:23:15.709110 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:41083
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:41083
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/testchainca.cert with env {}
W20260928 06:23:15.794389 5554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:15.794652 5554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:15.794693 5554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:15.795962 5554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:15.796038 5554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:15.796063 5554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:15.796097 5554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:15.796146 5554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:23:15.796177 5554 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:23:15.796203 5554 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 06:23:15.797760 5554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:41083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:41083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:15.798077 5554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:15.798314 5554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:15.803553 5560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:15.803658 5554 server_base.cc:1034] running on GCE node
W20260928 06:23:15.803562 5559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:15.803557 5562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:15.804073 5554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:15.804419 5554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:15.805583 5554 hybrid_clock.cc:648] HybridClock initialized: now 1790576595805562 us; error 43 us; skew 500 ppm
I20260928 06:23:15.807125 5554 webserver.cc:460] Webserver started at http://127.31.214.126:45497/ using document root <none> and password file <none>
I20260928 06:23:15.807431 5554 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:15.807484 5554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:15.807658 5554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:15.808805 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "e700b9048e9c4fcabf2c80f1548e2ee2"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "71fef5325baccec3578de9680c95fc72"
server_key_iv: "a48f2a7e7fcbc35bc772fe8022e779c1"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.809171 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "e700b9048e9c4fcabf2c80f1548e2ee2"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "71fef5325baccec3578de9680c95fc72"
server_key_iv: "a48f2a7e7fcbc35bc772fe8022e779c1"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.810622 5554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:15.811491 5568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:15.811686 5554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:15.811800 5554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "e700b9048e9c4fcabf2c80f1548e2ee2"
format_stamp: "Formatted at 2026-09-28 06:23:15 on dist-test-slave-q03f"
server_key: "71fef5325baccec3578de9680c95fc72"
server_key_iv: "a48f2a7e7fcbc35bc772fe8022e779c1"
server_key_version: "encryptionkey@0"
I20260928 06:23:15.811884 5554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:15.822145 5554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:15.822461 5554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:15.822616 5554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:15.826761 5554 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:41083
I20260928 06:23:15.826795 5620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:41083 every 8 connection(s)
I20260928 06:23:15.827173 5554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:15.828503 5621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:15.830957 5621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2: Bootstrap starting.
I20260928 06:23:15.831574 5621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:15.831868 5621 log.cc:824] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:15.832628 5621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2: No bootstrap required, opened a new log
I20260928 06:23:15.833856 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 5554
I20260928 06:23:15.834041 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:15.834256 32601 external_mini_cluster.cc:1428] Setting key 5bd4df187186e4e97da7c34226bfd658
I20260928 06:23:15.834679 5621 raft_consensus.cc:348] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e700b9048e9c4fcabf2c80f1548e2ee2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41083 } }
I20260928 06:23:15.834905 5621 raft_consensus.cc:374] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:15.834954 5621 raft_consensus.cc:729] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e700b9048e9c4fcabf2c80f1548e2ee2, State: Initialized, Role: FOLLOWER
I20260928 06:23:15.835093 5621 consensus_queue.cc:260] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e700b9048e9c4fcabf2c80f1548e2ee2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41083 } }
I20260928 06:23:15.835183 5621 raft_consensus.cc:388] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:15.835206 5621 raft_consensus.cc:482] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:15.835278 5621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:15.835834 5621 raft_consensus.cc:504] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e700b9048e9c4fcabf2c80f1548e2ee2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41083 } }
I20260928 06:23:15.835916 5621 leader_election.cc:302] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e700b9048e9c4fcabf2c80f1548e2ee2; no voters:
I20260928 06:23:15.836102 5621 leader_election.cc:288] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:15.836268 5626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:15.836382 5626 raft_consensus.cc:686] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [term 1 LEADER]: Becoming Leader. State: Replica: e700b9048e9c4fcabf2c80f1548e2ee2, State: Running, Role: LEADER
I20260928 06:23:15.836398 5621 sys_catalog.cc:564] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:15.836465 5626 consensus_queue.cc:237] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e700b9048e9c4fcabf2c80f1548e2ee2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41083 } }
I20260928 06:23:15.836823 5626 sys_catalog.cc:455] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e700b9048e9c4fcabf2c80f1548e2ee2. Latest consensus state: current_term: 1 leader_uuid: "e700b9048e9c4fcabf2c80f1548e2ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e700b9048e9c4fcabf2c80f1548e2ee2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41083 } } }
I20260928 06:23:15.836913 5626 sys_catalog.cc:458] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:15.837074 5627 sys_catalog.cc:455] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e700b9048e9c4fcabf2c80f1548e2ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e700b9048e9c4fcabf2c80f1548e2ee2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41083 } } }
I20260928 06:23:15.837152 5627 sys_catalog.cc:458] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:15.837117 5634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:15.837777 5634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:15.840112 5634 catalog_manager.cc:1347] Generated new cluster ID: 1b2bf19cc93a4c358921f191e5ff450a
I20260928 06:23:15.840224 5634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:15.840266 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:15.834800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.31.214.126:41083)
0928 06:23:15.835036 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:15.835045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:15.835075 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:23:15.835202 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:15.835210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:15.835283 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:15.835441 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:15.835683 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.837002 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.838125 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.838604 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.838727 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.839499 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:15.839503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:15.839504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:23:15.839721 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:15.839764 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:23:15.839775 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:15.839855 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:15.839922 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:23:15.840004 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
I20260928 06:23:15.841233 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:23:15.853870 5634 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:15.854513 5634 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:15.868845 5634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e700b9048e9c4fcabf2c80f1548e2ee2: Generated new TSK 0
I20260928 06:23:15.869120 5634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:23:15.984858 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:15.877682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.31.214.126:41083)
0928 06:23:15.877754 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:15.877757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:15.877898 (+ 141us) server_negotiation.cc:407] Connection header received
0928 06:23:15.878029 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:15.878030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:15.878044 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:15.878126 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:23:15.878292 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.878812 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.880006 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.880200 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.880252 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.880452 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:23:15.984609 (+104157us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:23:15.984793 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42156: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 06:23:15.984961 5625 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42156: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 06:23:15.985188 5650 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:15.877626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:41083 (local address 127.0.0.1:42156)
0928 06:23:15.877799 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:15.877812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:15.877883 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:15.878148 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:15.878149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:15.878151 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:23:15.878221 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:15.878224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.878936 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.878937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:15.879925 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:15.879928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:15.880283 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:15.880284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:15.880347 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:15.880353 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:23:15.984805 (+104452us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:23:15.984940 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.214.126:41083: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":110,"threads_started":1}
I20260928 06:23:15.985801 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 5554
2026-09-28T06:23:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (304 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (96 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:16 dist-test-slave-q03f krb5kdc[5662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:16 dist-test-slave-q03f krb5kdc[5662](info): set up 2 sockets
Sep 28 06:23:16 dist-test-slave-q03f krb5kdc[5662](info): commencing operation
krb5kdc: starting...
W20260928 06:23:17.238369 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:17 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:23:17.365676 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38779
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:17.448689 5675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:17.448910 5675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:17.448933 5675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:17.450171 5675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:17.450215 5675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:17.450230 5675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:17.450243 5675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:17.450255 5675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:17.451851 5675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:17.452126 5675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:17.452373 5675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:17.457587 5681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:17.457628 5683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:17.457573 5680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:17.457748 5675 server_base.cc:1034] running on GCE node
I20260928 06:23:17.458103 5675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:17.458415 5675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:17.459571 5675 hybrid_clock.cc:648] HybridClock initialized: now 1790576597459550 us; error 37 us; skew 500 ppm
Sep 28 06:23:17 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576597, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:17.462306 5675 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:23:17.462836 5675 webserver.cc:460] Webserver started at http://127.31.214.126:41875/ using document root <none> and password file <none>
I20260928 06:23:17.463073 5675 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:17.463141 5675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:17.463304 5675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:17.464448 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "4c66621fb0cc4a90868faab45a970b4c"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "ad44054b1e2e3cf1338e2d0e7796174f"
server_key_iv: "69476c2b4cc672b7ba6817fe168943e5"
server_key_version: "encryptionkey@0"
I20260928 06:23:17.464828 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "4c66621fb0cc4a90868faab45a970b4c"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "ad44054b1e2e3cf1338e2d0e7796174f"
server_key_iv: "69476c2b4cc672b7ba6817fe168943e5"
server_key_version: "encryptionkey@0"
I20260928 06:23:17.466243 5675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:23:17.467008 5690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:17.467190 5675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:17.467303 5675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "4c66621fb0cc4a90868faab45a970b4c"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "ad44054b1e2e3cf1338e2d0e7796174f"
server_key_iv: "69476c2b4cc672b7ba6817fe168943e5"
server_key_version: "encryptionkey@0"
I20260928 06:23:17.467381 5675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:17.483582 5675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:17.485678 5675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:17.485875 5675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:17.490356 5675 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:38779
I20260928 06:23:17.490396 5742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:38779 every 8 connection(s)
I20260928 06:23:17.490820 5675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:17.491932 5743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:17.495047 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c: Bootstrap starting.
I20260928 06:23:17.495711 5743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:17.496047 5743 log.cc:824] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:17.496837 5743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c: No bootstrap required, opened a new log
I20260928 06:23:17.498807 5743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c66621fb0cc4a90868faab45a970b4c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38779 } }
I20260928 06:23:17.499003 5743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:17.499058 5743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c66621fb0cc4a90868faab45a970b4c, State: Initialized, Role: FOLLOWER
I20260928 06:23:17.499245 5743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c66621fb0cc4a90868faab45a970b4c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38779 } }
I20260928 06:23:17.499354 5743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:17.499404 5743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:17.499460 5743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:17.500073 5743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c66621fb0cc4a90868faab45a970b4c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38779 } }
I20260928 06:23:17.500211 5743 leader_election.cc:302] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c66621fb0cc4a90868faab45a970b4c; no voters:
I20260928 06:23:17.500416 5743 leader_election.cc:288] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:17.500557 5746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:17.500713 5743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:17.500962 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 5675
I20260928 06:23:17.501013 5746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [term 1 LEADER]: Becoming Leader. State: Replica: 4c66621fb0cc4a90868faab45a970b4c, State: Running, Role: LEADER
I20260928 06:23:17.501088 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:17.501114 5746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c66621fb0cc4a90868faab45a970b4c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38779 } }
I20260928 06:23:17.501288 32601 external_mini_cluster.cc:1428] Setting key 876e2f61340416db19a407245dbc3d65
I20260928 06:23:17.502199 5747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c66621fb0cc4a90868faab45a970b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c66621fb0cc4a90868faab45a970b4c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38779 } } }
I20260928 06:23:17.502302 5747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:17.502346 5748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c66621fb0cc4a90868faab45a970b4c. Latest consensus state: current_term: 1 leader_uuid: "4c66621fb0cc4a90868faab45a970b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c66621fb0cc4a90868faab45a970b4c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38779 } } }
I20260928 06:23:17.502674 5748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c [sys.catalog]: This master's current role is: LEADER
W20260928 06:23:17.502908 5763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:23:17.502972 5763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:23:17.503065 5764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:17.503325 5764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:17.504938 5764 catalog_manager.cc:1347] Generated new cluster ID: 723a064cd33c416590ba4d0cd8f1f269
I20260928 06:23:17.504999 5764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:17.508936 5764 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:17.509718 5764 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 06:23:17 dist-test-slave-q03f krb5kdc[5662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:17.514518 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:17.501851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.31.214.126:38779)
0928 06:23:17.502288 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:17.502300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:17.502659 (+ 359us) server_negotiation.cc:407] Connection header received
0928 06:23:17.502886 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:17.502894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:17.502968 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:17.503184 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:17.504000 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.505286 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.506458 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.507183 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.507314 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:17.510387 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:17.510392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:17.510393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:17.510407 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:17.512663 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:17.513121 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:17.513125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:17.513126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:17.513251 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:17.513481 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:17.513482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:17.513482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:17.513799 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:17.514020 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:17.514108 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:23:17.514267 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":115,"threads_started":1}
I20260928 06:23:17.519301 5764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c66621fb0cc4a90868faab45a970b4c: Generated new TSK 0
I20260928 06:23:17.519523 5764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:23:17.553076 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38779
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:17.639051 5769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:17.639309 5769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:17.639353 5769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:17.640599 5769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:17.640672 5769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:17.640720 5769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:17.642405 5769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:17.642686 5769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:17.642946 5769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:17.648996 5775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:17.648978 5777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:17.649067 5769 server_base.cc:1034] running on GCE node
W20260928 06:23:17.648978 5774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:17.649396 5769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:17.649658 5769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:17.650847 5769 hybrid_clock.cc:648] HybridClock initialized: now 1790576597650834 us; error 51 us; skew 500 ppm
Sep 28 06:23:17 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576597, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:17.653703 5769 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:23:17.654238 5769 webserver.cc:460] Webserver started at http://127.31.214.65:37067/ using document root <none> and password file <none>
I20260928 06:23:17.654475 5769 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:17.654544 5769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:17.654706 5769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:17.655855 5769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "6349d5df79d3492a9f2a23a8c5ed9ac2"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "a06aa50e32b8992d5169ce101c96bed9"
server_key_iv: "89ccece01308f4424ae52757b11c0ef6"
server_key_version: "encryptionkey@0"
I20260928 06:23:17.656235 5769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "6349d5df79d3492a9f2a23a8c5ed9ac2"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "a06aa50e32b8992d5169ce101c96bed9"
server_key_iv: "89ccece01308f4424ae52757b11c0ef6"
server_key_version: "encryptionkey@0"
I20260928 06:23:17.657650 5769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:17.658676 5784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:17.658984 5769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:17.659117 5769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "6349d5df79d3492a9f2a23a8c5ed9ac2"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "a06aa50e32b8992d5169ce101c96bed9"
server_key_iv: "89ccece01308f4424ae52757b11c0ef6"
server_key_version: "encryptionkey@0"
I20260928 06:23:17.659209 5769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:17.685580 5769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:17.687791 5769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:17.687976 5769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:17.688235 5769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:17.688570 5769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:17.688625 5769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:17.688691 5769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:17.688721 5769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:17.695142 5769 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:34897
I20260928 06:23:17.695187 5897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:34897 every 8 connection(s)
I20260928 06:23:17.695539 5769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:23:17.699203 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 5769
I20260928 06:23:17.699295 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:23:17.699528 32601 external_mini_cluster.cc:1428] Setting key 8a408f241892b3077b43e43a36bc94f3
Sep 28 06:23:17 dist-test-slave-q03f krb5kdc[5662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576597, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:17.707023 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:17.696307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:40593 (local address 127.31.214.126:38779)
0928 06:23:17.696387 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:17.696390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:17.696648 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:23:17.697587 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:17.697588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:17.697599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:17.697655 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:17.698439 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.699031 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.700443 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.700818 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.700887 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:17.703485 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:17.703487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:17.703487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:17.703496 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:17.705284 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:17.705731 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:17.705733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:17.705733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:17.705803 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:17.706196 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:17.706197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:17.706198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:17.706360 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:17.706442 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:17.706791 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 06:23:17.706901 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:23:17.707101 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:17.696287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38779 (local address 127.31.214.65:40593)
0928 06:23:17.696510 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:17.696551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:17.697487 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:17.697772 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:17.697777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:17.698127 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:17.698335 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:17.698340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.699149 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.699150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:17.700365 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:17.700368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.700941 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.700943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:17.701072 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:17.701709 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:17.701719 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:17.703387 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:17.705379 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:17.705388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:17.705391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:17.705641 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:17.705911 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:17.705913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:17.705913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:17.706112 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:17.706472 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:17.706484 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:17.706628 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 06:23:17.706722 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:23:17.706895 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":116,"threads_started":1}
I20260928 06:23:17.707700 5898 heartbeater.cc:347] Connected to a master server at 127.31.214.126:38779
I20260928 06:23:17.707801 5898 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:17.708014 5898 heartbeater.cc:511] Master 127.31.214.126:38779 requested a full tablet report, sending...
I20260928 06:23:17.708536 5707 ts_manager.cc:194] Registered new tserver with Master: 6349d5df79d3492a9f2a23a8c5ed9ac2 (127.31.214.65:34897)
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
I20260928 06:23:17.709215 5707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:40593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:23:17.730499 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38779
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:17.819449 5903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:17.819720 5903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:17.819762 5903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:17.820992 5903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:17.821080 5903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:17.821126 5903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:17.822798 5903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:17.823098 5903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:17.823348 5903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:17.828660 5909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:17.828658 5911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:17.829003 5908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:17.829183 5903 server_base.cc:1034] running on GCE node
I20260928 06:23:17.829459 5903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:17.829710 5903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:17.830869 5903 hybrid_clock.cc:648] HybridClock initialized: now 1790576597830847 us; error 38 us; skew 500 ppm
Sep 28 06:23:17 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576597, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:17.833525 5903 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:23:17.834095 5903 webserver.cc:460] Webserver started at http://127.31.214.66:35825/ using document root <none> and password file <none>
I20260928 06:23:17.834349 5903 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:17.834434 5903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:17.834580 5903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:17.835656 5903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "dbca9b61675c4ba9a21a7e4583ce7130"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "b5b6c2ddb733843715c06a9f269fbf0a"
server_key_iv: "244ba07f6a6d10fbe6b1fa7e3bb1f600"
server_key_version: "encryptionkey@0"
I20260928 06:23:17.836061 5903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "dbca9b61675c4ba9a21a7e4583ce7130"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "b5b6c2ddb733843715c06a9f269fbf0a"
server_key_iv: "244ba07f6a6d10fbe6b1fa7e3bb1f600"
server_key_version: "encryptionkey@0"
I20260928 06:23:17.837443 5903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:17.838372 5918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:17.838596 5903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:17.838709 5903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "dbca9b61675c4ba9a21a7e4583ce7130"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "b5b6c2ddb733843715c06a9f269fbf0a"
server_key_iv: "244ba07f6a6d10fbe6b1fa7e3bb1f600"
server_key_version: "encryptionkey@0"
I20260928 06:23:17.838789 5903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:17.851617 5903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:17.853870 5903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:17.854068 5903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:17.854324 5903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:17.854645 5903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:17.854707 5903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:17.854753 5903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:17.854782 5903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:17.861066 5903 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:37977
I20260928 06:23:17.861129 6031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:37977 every 8 connection(s)
I20260928 06:23:17.861444 5903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:23:17.866712 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 5903
I20260928 06:23:17.866825 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:17.867043 32601 external_mini_cluster.cc:1428] Setting key 9f9ce8f79d19ae1d3fea40b50cb59520
Sep 28 06:23:17 dist-test-slave-q03f krb5kdc[5662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576597, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:17.872895 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:17.862250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:33395 (local address 127.31.214.126:38779)
0928 06:23:17.862310 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:17.862311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:17.862554 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:23:17.863675 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:17.863676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:17.863691 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:17.863747 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:17.864420 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.864942 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.865996 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.866216 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.866305 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:17.869338 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:17.869340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:17.869340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:17.869354 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:17.871274 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:17.871707 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:17.871709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:17.871709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:17.871786 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:17.872118 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:17.872120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:17.872120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:17.872279 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:17.872354 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:17.872652 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 06:23:17.872749 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:23:17.872989 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:17.862239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38779 (local address 127.31.214.66:33395)
0928 06:23:17.862419 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:17.862465 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:17.863548 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:17.863828 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:17.863837 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:17.864115 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:17.864316 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:17.864321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.865046 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.865048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:17.865872 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:17.865875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:17.866316 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:17.866318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:17.866465 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:17.867243 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:17.867365 (+ 122us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:17.869163 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:17.871374 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:17.871381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:17.871383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:17.871614 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:17.871891 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:17.871892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:17.871892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:17.872034 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:17.872356 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:17.872360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:17.872495 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:23:17.872596 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:23:17.872736 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260928 06:23:17.873615 6032 heartbeater.cc:347] Connected to a master server at 127.31.214.126:38779
I20260928 06:23:17.873737 6032 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:17.873980 6032 heartbeater.cc:511] Master 127.31.214.126:38779 requested a full tablet report, sending...
I20260928 06:23:17.874485 5707 ts_manager.cc:194] Registered new tserver with Master: dbca9b61675c4ba9a21a7e4583ce7130 (127.31.214.66:37977)
I20260928 06:23:17.875068 5707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:33395
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:23:17.898211 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38779
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:17.983032 6037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:17.983304 6037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:17.983353 6037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:17.984627 6037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:17.984709 6037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:17.984757 6037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:17.986611 6037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:17.986928 6037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:17.987185 6037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:17.992694 6045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:17.992688 6042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:17.992694 6043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:17.992997 6037 server_base.cc:1034] running on GCE node
I20260928 06:23:17.993325 6037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:17.993597 6037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:17.994776 6037 hybrid_clock.cc:648] HybridClock initialized: now 1790576597994764 us; error 38 us; skew 500 ppm
Sep 28 06:23:17 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576597, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:17.997450 6037 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:23:17.998203 6037 webserver.cc:460] Webserver started at http://127.31.214.67:38795/ using document root <none> and password file <none>
I20260928 06:23:17.998466 6037 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:17.998538 6037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:17.998685 6037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:18.000250 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "92553c7f033d4afbb784f87cfed779e5"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "0c33d22f2d6168769d2dcd04b09af440"
server_key_iv: "b677e2f8a4757d17e8d98e098628949b"
server_key_version: "encryptionkey@0"
I20260928 06:23:18.000677 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "92553c7f033d4afbb784f87cfed779e5"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "0c33d22f2d6168769d2dcd04b09af440"
server_key_iv: "b677e2f8a4757d17e8d98e098628949b"
server_key_version: "encryptionkey@0"
I20260928 06:23:18.005373 6037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 06:23:18.006508 6052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:18.006699 6037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:23:18.006807 6037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "92553c7f033d4afbb784f87cfed779e5"
format_stamp: "Formatted at 2026-09-28 06:23:17 on dist-test-slave-q03f"
server_key: "0c33d22f2d6168769d2dcd04b09af440"
server_key_iv: "b677e2f8a4757d17e8d98e098628949b"
server_key_version: "encryptionkey@0"
I20260928 06:23:18.006897 6037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:18.022908 6037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:18.025130 6037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:18.025249 6037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:18.025460 6037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:18.025731 6037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:18.025766 6037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:18.025789 6037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:18.025801 6037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:18.032011 6037 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:44113
I20260928 06:23:18.032063 6165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:44113 every 8 connection(s)
I20260928 06:23:18.032433 6037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:18.034085 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 6037
I20260928 06:23:18.034188 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:18.034386 32601 external_mini_cluster.cc:1428] Setting key 2619f805074b425cb707e72e9ab0de6a
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576597, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:18.042738 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:18.033295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:39341 (local address 127.31.214.126:38779)
0928 06:23:18.033371 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:18.033373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:18.033584 (+ 211us) server_negotiation.cc:407] Connection header received
0928 06:23:18.034889 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:18.034890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:18.034903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:18.034966 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:18.035792 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.036319 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.037389 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.037599 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.037660 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:18.039878 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:18.039880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:18.039880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:18.039887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:18.041189 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:18.041583 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:18.041583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:18.041584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:18.041647 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:18.042118 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:18.042120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:18.042120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:18.042235 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:18.042292 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:18.042594 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 06:23:18.042669 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:23:18.042918 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:18.033274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38779 (local address 127.31.214.67:39341)
0928 06:23:18.033450 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:18.033493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:18.034793 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:18.035050 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:18.035055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:18.035457 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:18.035678 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:18.035682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.036449 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.036449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:18.037272 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:18.037274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.037679 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.037679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:18.037782 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:18.038317 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:18.038326 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:18.039791 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:18.041264 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:18.041270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:18.041272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:18.041506 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:18.041811 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:18.041812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:18.041813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:18.042032 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:18.042306 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:18.042320 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:18.042447 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 06:23:18.042546 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 06:23:18.042685 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260928 06:23:18.043401 6166 heartbeater.cc:347] Connected to a master server at 127.31.214.126:38779
I20260928 06:23:18.043485 6166 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:18.043637 6166 heartbeater.cc:511] Master 127.31.214.126:38779 requested a full tablet report, sending...
I20260928 06:23:18.044035 5707 ts_manager.cc:194] Registered new tserver with Master: 92553c7f033d4afbb784f87cfed779e5 (127.31.214.67:44113)
I20260928 06:23:18.044415 5707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:39341
I20260928 06:23:18.046381 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/test-admin.keytab.
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:18.062688 32601 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 06:23:18.064917 32601 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5cc' of type FILE
W20260928 06:23:18.064924 6173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:18.063979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.31.214.65:34897)
0928 06:23:18.064183 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:18.064188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:18.064200 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:23:18.064227 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:18.064229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:18.064244 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:18.064309 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:18.064811 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38386: BlockingRecv error: recv got EOF from 127.0.0.1:38386 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:18.066157 32601 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:18.067210 32601 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:23:18.075584 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:18.067847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38388 (local address 127.31.214.65:34897)
0928 06:23:18.067924 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:18.067926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:18.067941 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:23:18.068012 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:18.068013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:18.068023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:18.068073 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:18.068505 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.069238 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.070006 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.070464 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.070541 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:18.072815 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:18.072823 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:18.072823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:18.072829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:18.074443 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:18.074769 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:18.074771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:18.074772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:18.074832 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:18.075017 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:18.075017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:18.075017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:18.075146 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:18.075250 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:18.075350 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:23:18.075468 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 06:23:18.077782 6173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:18.077165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.31.214.65:34897)
0928 06:23:18.077211 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:18.077213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:18.077259 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:23:18.077366 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:18.077367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:18.077376 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:18.077430 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:18.077725 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38404: BlockingRecv error: recv got EOF from 127.0.0.1:38404 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:23:18.077823 32601 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:18.078838 32601 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:18.079689 32601 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:23:18.086594 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:18.080186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.31.214.65:34897)
0928 06:23:18.080235 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:18.080237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:18.080248 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:23:18.080305 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:18.080306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:18.080313 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:18.080356 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:18.080684 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.081176 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.081909 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.082169 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.082225 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:18.084182 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:18.084183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:18.084183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:18.084190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:18.085598 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:18.085909 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:18.085910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:18.085910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:18.086035 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:18.086217 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:18.086217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:18.086217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:18.086307 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:18.086401 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:18.086468 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:23:18.086532 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:23:18.088258 6173 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:18.087713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.31.214.65:34897)
0928 06:23:18.087752 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:18.087754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:18.087843 (+ 89us) server_negotiation.cc:407] Connection header received
0928 06:23:18.087860 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:18.087861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:18.087868 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:18.087913 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:18.088211 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38430: BlockingRecv error: recv got EOF from 127.0.0.1:38430 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:23:18.088392 32601 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790576570121568-32601-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:18.089412 32601 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:18.090287 32601 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[5662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:23:18.097402 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:18.090943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.31.214.65:34897)
0928 06:23:18.091028 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:18.091030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:18.091087 (+ 57us) server_negotiation.cc:407] Connection header received
0928 06:23:18.091132 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:18.091132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:18.091140 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:18.091185 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:18.091576 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.092050 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.092753 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:18.092935 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:18.092991 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:18.095108 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:18.095109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:18.095110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:18.095116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:18.096568 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:18.096855 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:18.096856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:18.096856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:18.096903 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:18.097045 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:18.097045 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:18.097046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:18.097120 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:18.097181 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:18.097280 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:23:18.097338 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:23:18.098067 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 5769
I20260928 06:23:18.104286 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 5903
I20260928 06:23:18.108768 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 6037
I20260928 06:23:18.112957 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 5675
2026-09-28T06:23:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1940 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[6186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[6186](info): set up 2 sockets
Sep 28 06:23:18 dist-test-slave-q03f krb5kdc[6186](info): commencing operation
krb5kdc: starting...
W20260928 06:23:19.162532 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:19 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.126.keytab.
Entry for principal oryx/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.126.keytab.
I20260928 06:23:19.290905 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:46093
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.126.keytab
--principal=oryx/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:46093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:19.374571 6199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:19.374790 6199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:19.374812 6199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:19.375927 6199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:19.375963 6199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:19.375977 6199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:19.375990 6199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:19.376003 6199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:19.377442 6199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:46093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.126.keytab
--principal=oryx/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:46093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:19.377648 6199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:19.377825 6199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:19.382987 6204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:19.383002 6207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:19.382980 6205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:19.383315 6199 server_base.cc:1034] running on GCE node
I20260928 06:23:19.383596 6199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:19.383912 6199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:19.385116 6199 hybrid_clock.cc:648] HybridClock initialized: now 1790576599385059 us; error 95 us; skew 500 ppm
Sep 28 06:23:19 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:19.395154 6199 init.cc:377] Logged in from keytab as oryx/127.31.214.126@KRBTEST.COM (short username oryx)
I20260928 06:23:19.395680 6199 webserver.cc:460] Webserver started at http://127.31.214.126:46461/ using document root <none> and password file <none>
I20260928 06:23:19.395943 6199 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:19.396030 6199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:19.396211 6199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:19.397312 6199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "8d549ccb5ce04c039d00ed910205c763"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "19b23a298f7f0ea2f138b3105fa3654b"
server_key_iv: "c521c6d1a8daaf3c9bd1bf1aa4cddc3c"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.397635 6199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "8d549ccb5ce04c039d00ed910205c763"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "19b23a298f7f0ea2f138b3105fa3654b"
server_key_iv: "c521c6d1a8daaf3c9bd1bf1aa4cddc3c"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.399076 6199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:19.399920 6214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.400213 6199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:19.400292 6199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "8d549ccb5ce04c039d00ed910205c763"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "19b23a298f7f0ea2f138b3105fa3654b"
server_key_iv: "c521c6d1a8daaf3c9bd1bf1aa4cddc3c"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.400346 6199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:19.411886 6199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:19.414346 6199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:19.414459 6199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:19.418431 6199 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:46093
I20260928 06:23:19.418500 6266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:46093 every 8 connection(s)
I20260928 06:23:19.418856 6199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:19.420032 6267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:19.422701 6267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763: Bootstrap starting.
I20260928 06:23:19.423368 6267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:19.423699 6267 log.cc:824] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:19.424474 6267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763: No bootstrap required, opened a new log
I20260928 06:23:19.426409 6267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } }
I20260928 06:23:19.426546 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 6199
I20260928 06:23:19.426587 6267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:19.426651 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:19.426640 6267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d549ccb5ce04c039d00ed910205c763, State: Initialized, Role: FOLLOWER
I20260928 06:23:19.426820 6267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } }
I20260928 06:23:19.426870 32601 external_mini_cluster.cc:1428] Setting key 33981003a5552488db12993a75894f61
I20260928 06:23:19.426918 6267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:19.426963 6267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:19.427021 6267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:19.428382 6267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } }
I20260928 06:23:19.428484 6267 leader_election.cc:302] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d549ccb5ce04c039d00ed910205c763; no voters:
I20260928 06:23:19.428681 6267 leader_election.cc:288] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:19.428730 6272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:19.428939 6272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 1 LEADER]: Becoming Leader. State: Replica: 8d549ccb5ce04c039d00ed910205c763, State: Running, Role: LEADER
I20260928 06:23:19.429039 6267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:19.429049 6272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } }
I20260928 06:23:19.429415 6272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d549ccb5ce04c039d00ed910205c763. Latest consensus state: current_term: 1 leader_uuid: "8d549ccb5ce04c039d00ed910205c763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } }
I20260928 06:23:19.429500 6272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:19.429426 6274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d549ccb5ce04c039d00ed910205c763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } }
I20260928 06:23:19.429842 6274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:19.429724 6277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:19.430619 6277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:19.432498 6277 catalog_manager.cc:1347] Generated new cluster ID: 2a770ff9ca0e4473b854b1176c2566bb
I20260928 06:23:19.432571 6277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:23:19 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:19.438243 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:19.427461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.31.214.126:46093)
0928 06:23:19.427689 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:19.427702 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:19.427750 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:23:19.427923 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:19.427931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:19.427992 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:19.428163 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:19.429062 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.430643 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.431466 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.432126 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.432238 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:19.434776 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:19.434780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:19.434780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:19.434793 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:19.436568 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:19.436955 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:19.436958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:19.436959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:19.437065 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:19.437296 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:19.437297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:19.437297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:19.437558 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:19.437747 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:19.437816 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:23:19.438085 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260928 06:23:19.464008 6277 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:19.464792 6277 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:19.473220 6277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763: Generated new TSK 0
I20260928 06:23:19.473462 6277 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab.
Entry for principal oryx/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab.
I20260928 06:23:19.509461 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:46093
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab
--principal=oryx/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:19.594563 6293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:19.594817 6293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:19.594858 6293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:19.596067 6293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:19.596140 6293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:19.596186 6293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:19.597746 6293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab
--principal=oryx/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:46093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:19.598060 6293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:19.598306 6293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:19.604089 6298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:19.604101 6299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:19.604089 6301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:19.604426 6293 server_base.cc:1034] running on GCE node
I20260928 06:23:19.604653 6293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:19.604923 6293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:19.606110 6293 hybrid_clock.cc:648] HybridClock initialized: now 1790576599606104 us; error 38 us; skew 500 ppm
Sep 28 06:23:19 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:19.609014 6293 init.cc:377] Logged in from keytab as oryx/127.31.214.65@KRBTEST.COM (short username oryx)
I20260928 06:23:19.609572 6293 webserver.cc:460] Webserver started at http://127.31.214.65:45091/ using document root <none> and password file <none>
I20260928 06:23:19.609830 6293 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:19.609903 6293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:19.610077 6293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:19.611191 6293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "edcac2b2bcd1470ba1bd11588faec84b"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "c068212924441fe9a4fc4e6600ee5e3a"
server_key_iv: "cb85d0f97f5e6ed56270e634236b49c4"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.611569 6293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "edcac2b2bcd1470ba1bd11588faec84b"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "c068212924441fe9a4fc4e6600ee5e3a"
server_key_iv: "cb85d0f97f5e6ed56270e634236b49c4"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.613003 6293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:23:19.614007 6308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.614202 6293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:19.614403 6293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "edcac2b2bcd1470ba1bd11588faec84b"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "c068212924441fe9a4fc4e6600ee5e3a"
server_key_iv: "cb85d0f97f5e6ed56270e634236b49c4"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.614490 6293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:19.648216 6293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:19.650547 6293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:19.650718 6293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:19.650980 6293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:19.651338 6293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:19.651391 6293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.651428 6293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:19.651472 6293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.657500 6293 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:36869
I20260928 06:23:19.657562 6421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:36869 every 8 connection(s)
I20260928 06:23:19.657915 6293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:23:19 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:19.665360 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 6293
I20260928 06:23:19.665452 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:23:19.665693 32601 external_mini_cluster.cc:1428] Setting key ea420b030e6e35c38ed6644c2ac47410
I20260928 06:23:19.668654 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:19.658951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:51047 (local address 127.31.214.126:46093)
0928 06:23:19.659038 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:19.659040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:19.659227 (+ 187us) server_negotiation.cc:407] Connection header received
0928 06:23:19.659771 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:19.659772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:19.659786 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:19.659851 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:19.660836 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.661377 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.662454 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.662660 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.662721 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:19.665126 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:19.665127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:19.665128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:19.665136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:19.666971 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:19.667518 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:19.667519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:19.667520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:19.667581 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:19.667903 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:19.667904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:19.667905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:19.668071 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:19.668147 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:19.668429 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 06:23:19.668532 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:23:19.668681 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:19.658889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:51047)
0928 06:23:19.659086 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:19.659131 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:19.659650 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:19.659927 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:19.659931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:19.660512 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:19.660718 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:19.660723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.661487 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.661488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:19.662374 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:19.662377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.662742 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.662742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:19.662869 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:19.663464 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:19.663473 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:19.665006 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:19.667068 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:19.667076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:19.667078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:19.667424 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:19.667656 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:19.667657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:19.667658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:19.667820 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:19.668148 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:19.668154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:19.668292 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:23:19.668381 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:23:19.668511 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
I20260928 06:23:19.669344 6422 heartbeater.cc:347] Connected to a master server at 127.31.214.126:46093
I20260928 06:23:19.669468 6422 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:19.669677 6422 heartbeater.cc:511] Master 127.31.214.126:46093 requested a full tablet report, sending...
I20260928 06:23:19.670341 6231 ts_manager.cc:194] Registered new tserver with Master: edcac2b2bcd1470ba1bd11588faec84b (127.31.214.65:36869)
I20260928 06:23:19.671378 6231 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:51047
WARNING: no policy specified for oryx/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab.
Entry for principal oryx/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab.
I20260928 06:23:19.694571 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:46093
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab
--principal=oryx/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:19.780185 6427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:19.780506 6427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:19.780535 6427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:19.781747 6427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:19.781790 6427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:19.781819 6427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:19.783306 6427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab
--principal=oryx/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:46093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:19.783532 6427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:19.783720 6427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:19.789098 6432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:19.789105 6435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:19.789130 6433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:19.789531 6427 server_base.cc:1034] running on GCE node
I20260928 06:23:19.789717 6427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:19.789932 6427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:19.791184 6427 hybrid_clock.cc:648] HybridClock initialized: now 1790576599791159 us; error 42 us; skew 500 ppm
Sep 28 06:23:19 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:19.794232 6427 init.cc:377] Logged in from keytab as oryx/127.31.214.66@KRBTEST.COM (short username oryx)
I20260928 06:23:19.794780 6427 webserver.cc:460] Webserver started at http://127.31.214.66:42077/ using document root <none> and password file <none>
I20260928 06:23:19.795051 6427 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:19.795125 6427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:19.795363 6427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:19.796561 6427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "d8e4de63f2044696abbf32f9687fb04c"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "5b83b16b7e04ecfe0804b2e24b52abb6"
server_key_iv: "fd33a85d3d8b393bbb138bcbf9574ab4"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.796948 6427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "d8e4de63f2044696abbf32f9687fb04c"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "5b83b16b7e04ecfe0804b2e24b52abb6"
server_key_iv: "fd33a85d3d8b393bbb138bcbf9574ab4"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.798475 6427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:19.799270 6442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.799506 6427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:19.799609 6427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "d8e4de63f2044696abbf32f9687fb04c"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "5b83b16b7e04ecfe0804b2e24b52abb6"
server_key_iv: "fd33a85d3d8b393bbb138bcbf9574ab4"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.799695 6427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:19.808223 6427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:19.810525 6427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:19.810683 6427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:19.810922 6427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:19.811239 6427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:19.811298 6427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.811365 6427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:19.811395 6427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.817459 6427 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:43369
I20260928 06:23:19.817545 6555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:43369 every 8 connection(s)
I20260928 06:23:19.817907 6427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:23:19.819926 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 6427
I20260928 06:23:19.820052 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:19.820273 32601 external_mini_cluster.cc:1428] Setting key 71a99b41542ec6d4222e98c86178819c
Sep 28 06:23:19 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:19.829414 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:19.818795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:33081 (local address 127.31.214.126:46093)
0928 06:23:19.818880 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:19.818882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:19.819028 (+ 146us) server_negotiation.cc:407] Connection header received
0928 06:23:19.819513 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:19.819514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:19.819524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:19.819595 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:19.820867 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.821495 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.823009 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.823236 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.823334 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:19.826090 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:19.826092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:19.826093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:19.826102 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:19.827847 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:19.828272 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:19.828273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:19.828274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:19.828343 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:19.828669 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:19.828670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:19.828671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:19.828835 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:19.828917 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:19.829194 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:23:19.829289 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:23:19.829556 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:19.818707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:33081)
0928 06:23:19.818894 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:19.818938 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:19.819426 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:19.819651 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:19.819654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:19.820533 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:19.820766 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:19.820773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.821604 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.821605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:19.822910 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:19.822914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.823345 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.823346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:19.823485 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:19.824063 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:19.824071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:19.825885 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:19.827943 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:19.827950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:19.827952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:19.828187 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:19.828417 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:19.828417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:19.828418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:19.828596 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:19.828912 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:19.828918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:19.829050 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 06:23:19.829139 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:23:19.829283 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.31.214.67@KRBTEST.COM; defaulting to no policy
I20260928 06:23:19.830173 6556 heartbeater.cc:347] Connected to a master server at 127.31.214.126:46093
I20260928 06:23:19.830288 6556 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:19.830554 6556 heartbeater.cc:511] Master 127.31.214.126:46093 requested a full tablet report, sending...
I20260928 06:23:19.831102 6231 ts_manager.cc:194] Registered new tserver with Master: d8e4de63f2044696abbf32f9687fb04c (127.31.214.66:43369)
I20260928 06:23:19.831682 6231 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:33081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab.
Entry for principal oryx/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab.
I20260928 06:23:19.849085 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:46093
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab
--principal=oryx/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:19.933764 6561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:19.934048 6561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:19.934100 6561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:19.935287 6561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:19.935328 6561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:19.935353 6561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:19.936786 6561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab
--principal=oryx/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:46093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:19.937042 6561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:19.937297 6561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:19.942750 6567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:19.942744 6566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:19.942801 6569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:19.943063 6561 server_base.cc:1034] running on GCE node
I20260928 06:23:19.943362 6561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:19.943603 6561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:19.944775 6561 hybrid_clock.cc:648] HybridClock initialized: now 1790576599944742 us; error 65 us; skew 500 ppm
Sep 28 06:23:19 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:19.947564 6561 init.cc:377] Logged in from keytab as oryx/127.31.214.67@KRBTEST.COM (short username oryx)
I20260928 06:23:19.948202 6561 webserver.cc:460] Webserver started at http://127.31.214.67:43309/ using document root <none> and password file <none>
I20260928 06:23:19.948470 6561 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:19.948518 6561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:19.948616 6561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:19.949723 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "ff725a15eb8f429eb9c4c76a8c46dafb"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "a0d1442f33b55a8ffc363af75747cac7"
server_key_iv: "e45aef843a44d06ac11be99bd824b8e5"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.950136 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "ff725a15eb8f429eb9c4c76a8c46dafb"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "a0d1442f33b55a8ffc363af75747cac7"
server_key_iv: "e45aef843a44d06ac11be99bd824b8e5"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.951594 6561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:19.952446 6576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.952636 6561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:19.952757 6561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "ff725a15eb8f429eb9c4c76a8c46dafb"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "a0d1442f33b55a8ffc363af75747cac7"
server_key_iv: "e45aef843a44d06ac11be99bd824b8e5"
server_key_version: "encryptionkey@0"
I20260928 06:23:19.952842 6561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:19.967561 6561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:19.969863 6561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:19.970084 6561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:19.970366 6561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:19.970712 6561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:19.970770 6561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.970814 6561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:19.970842 6561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:19.977773 6561 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:42441
I20260928 06:23:19.977818 6689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:42441 every 8 connection(s)
I20260928 06:23:19.978271 6561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:19.984670 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 6561
I20260928 06:23:19.984776 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:19.985014 32601 external_mini_cluster.cc:1428] Setting key 8afb6e05199f70a5d61c10dd7d6de0ed
Sep 28 06:23:19 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:19.989391 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:19.979247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:34471 (local address 127.31.214.126:46093)
0928 06:23:19.979346 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:19.979349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:19.979603 (+ 254us) server_negotiation.cc:407] Connection header received
0928 06:23:19.980102 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:19.980104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:19.980115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:19.980183 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:19.981271 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.981864 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.983027 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.983220 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.983282 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:19.986141 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:19.986143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:19.986143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:19.986152 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:19.987669 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:19.988219 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:19.988221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:19.988221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:19.988294 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:19.988612 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:19.988613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:19.988613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:19.988748 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:19.988809 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:19.989234 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 06:23:19.989320 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:23:19.989538 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:19.979161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:34471)
0928 06:23:19.979420 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:19.979466 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:19.979983 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:19.980249 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:19.980253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:19.980952 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:19.981159 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:19.981165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.981998 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.981999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:19.982939 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:19.982942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:19.983328 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:19.983330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:19.983444 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:19.984048 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:19.984057 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:19.986046 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:19.987772 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:19.987780 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:19.987782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:19.988023 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:19.988397 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:19.988398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:19.988398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:19.988520 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:19.988854 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:19.988861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:19.988999 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:23:19.989104 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:23:19.989295 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":124,"threads_started":1}
I20260928 06:23:19.990180 6690 heartbeater.cc:347] Connected to a master server at 127.31.214.126:46093
I20260928 06:23:19.990274 6690 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:19.990442 6690 heartbeater.cc:511] Master 127.31.214.126:46093 requested a full tablet report, sending...
I20260928 06:23:19.990811 6231 ts_manager.cc:194] Registered new tserver with Master: ff725a15eb8f429eb9c4c76a8c46dafb (127.31.214.67:42441)
I20260928 06:23:19.991240 6231 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:34471
I20260928 06:23:19.997270 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.214.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.125.keytab.
Entry for principal oryx/127.31.214.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.125.keytab.
I20260928 06:23:20.030616 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.125:34981
--webserver_interface=127.31.214.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.125.keytab
--principal=oryx/127.31.214.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:46093,127.31.214.125:34981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:20.115892 6695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:20.116142 6695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:20.116180 6695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:20.117439 6695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:20.117504 6695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:20.117530 6695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:20.117561 6695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:20.117587 6695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:20.119279 6695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:46093,127.31.214.125:34981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.125.keytab
--principal=oryx/127.31.214.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.125:34981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.214.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
W20260928 06:23:20.119458 6695 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.214.126:46093,127.31.214.125:34981'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 06:23:20.119618 6695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:20.119854 6695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:20.125051 6700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.125061 6701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.125134 6695 server_base.cc:1034] running on GCE node
W20260928 06:23:20.125063 6703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.125517 6695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:20.125883 6695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:20.127039 6695 hybrid_clock.cc:648] HybridClock initialized: now 1790576600127013 us; error 39 us; skew 500 ppm
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:20.129997 6695 init.cc:377] Logged in from keytab as oryx/127.31.214.125@KRBTEST.COM (short username oryx)
I20260928 06:23:20.130541 6695 webserver.cc:460] Webserver started at http://127.31.214.125:44433/ using document root <none> and password file <none>
I20260928 06:23:20.130745 6695 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:20.130787 6695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:20.130891 6695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:20.131932 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data/instance:
uuid: "b0014cf7b5634653bacf2358884671a6"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "729bccd30b8e342ab74cfca128f094c1"
server_key_iv: "4dbe4e351d140defae41c2aa357284b2"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.132243 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal/instance:
uuid: "b0014cf7b5634653bacf2358884671a6"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "729bccd30b8e342ab74cfca128f094c1"
server_key_iv: "4dbe4e351d140defae41c2aa357284b2"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.133574 6695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:20.134434 6710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.134634 6695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:23:20.134707 6695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal
uuid: "b0014cf7b5634653bacf2358884671a6"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "729bccd30b8e342ab74cfca128f094c1"
server_key_iv: "4dbe4e351d140defae41c2aa357284b2"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.134761 6695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:20.150913 6695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:20.153362 6695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:20.153477 6695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:20.157819 6695 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.125:34981
I20260928 06:23:20.157852 6762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.125:34981 every 8 connection(s)
I20260928 06:23:20.158229 6695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
I20260928 06:23:20.159406 6763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:20.161167 6763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:23:20.166462 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 6695
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.125@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:20.171478 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.162025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.31.214.126:46093)
0928 06:23:20.162100 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.162102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.162335 (+ 233us) server_negotiation.cc:407] Connection header received
0928 06:23:20.162857 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.162858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.162873 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.162937 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.163625 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.164139 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.165150 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.165351 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.165456 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.167900 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.167902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.167902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.167912 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.169795 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.170259 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.170261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.170261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.170334 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.170623 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.170624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.170624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.170805 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.170875 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.171205 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.171366 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:23:20.171504 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.162015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43042)
0928 06:23:20.162201 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.162238 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.162734 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.163016 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.163019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.163279 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.163518 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.163522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.164239 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.164240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.165053 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.165055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.165474 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.165475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.165602 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.166169 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.166179 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.167800 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.169877 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.169884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.169886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.170174 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.170408 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.170409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.170409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.170550 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.170878 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.170883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.171002 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.171136 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.171326 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260928 06:23:20.172335 6763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.214.125@KRBTEST.COM
I20260928 06:23:20.176421 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.167014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.31.214.125:34981)
0928 06:23:20.167331 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.167335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.167350 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:23:20.167367 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.167369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.167385 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.167467 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.168178 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.169180 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.170276 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.170718 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.170770 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.173394 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.173402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.173402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.173409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.175090 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.175507 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.175510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.175510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.175555 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.175776 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.175777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.175777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.175933 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.176115 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.176159 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.176243 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":90,"threads_started":1}
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.125@KRBTEST.COM for oryx/127.31.214.125@KRBTEST.COM
I20260928 06:23:20.180102 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.172591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:34981 (local address 127.0.0.1:52092)
0928 06:23:20.172640 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.172651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.172686 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.172958 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.172958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.173102 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.173142 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.173144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.174067 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.174068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.174721 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.174724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.175225 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.175226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.175288 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.175712 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.175717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.177035 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.179064 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.179064 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.179065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.179226 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.179443 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.179444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.179445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.179513 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.179763 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.179764 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.179843 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.179934 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.180039 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:23:20.180271 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.172674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.31.214.125:34981)
0928 06:23:20.172855 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.172856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.172862 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:23:20.172875 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.172876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.172883 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.172956 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.173216 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.173966 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.174813 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.175159 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.175227 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.177117 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.177119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.177119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.177126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.178985 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.179296 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.179297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.179297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.179346 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.179580 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.179580 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.179581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.179675 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.179728 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.179987 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.180093 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260928 06:23:20.181620 6763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: Bootstrap starting.
I20260928 06:23:20.182356 6763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:20.182643 6763 log.cc:824] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:20.183442 6763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: No bootstrap required, opened a new log
I20260928 06:23:20.184827 6763 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } }
I20260928 06:23:20.184949 6763 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:20.184973 6763 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0014cf7b5634653bacf2358884671a6, State: Initialized, Role: FOLLOWER
I20260928 06:23:20.185118 6763 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } }
I20260928 06:23:20.185385 6770 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } } }
I20260928 06:23:20.185469 6770 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:23:20.185608 6763 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 06:23:20.187386 6781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:23:20.187443 6781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:23:20.189544 6695 master_runner.cc:428] Detected that this master b0014cf7b5634653bacf2358884671a6 is joining an existing cluster
I20260928 06:23:20.189599 6695 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.214.125:34981
I20260928 06:23:20.198261 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.190691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:34981 (local address 127.0.0.1:52106)
0928 06:23:20.190865 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.190877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.190943 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.191215 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.191216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.191448 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.191516 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.191518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.192459 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.192460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.193075 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.193077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.193645 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.193646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.193703 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.194476 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.194482 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.194924 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.196934 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.196935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.196936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.197154 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.197386 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.197388 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.197388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.197468 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.197894 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.197895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.197966 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.198039 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.198141 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260928 06:23:20.198647 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.190815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.31.214.125:34981)
0928 06:23:20.190857 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.190860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.190945 (+ 85us) server_negotiation.cc:407] Connection header received
0928 06:23:20.191135 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.191136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.191144 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.191199 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.191602 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.192358 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.193271 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.193545 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.193628 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.195049 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.195051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.195051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.195058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.196847 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.197241 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.197242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.197242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.197302 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.197539 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.197540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.197540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.197674 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.197741 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.198475 (+ 734us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.198559 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:23:20.198700 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.190776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.31.214.126:46093)
0928 06:23:20.190863 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.190865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.190929 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:23:20.191000 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.191001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.191012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.191085 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.192476 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.193241 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.194372 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.194614 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.194732 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.195817 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.195819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.195819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.195828 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.197398 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.197747 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.197748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.197748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.197809 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.198050 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.198051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.198052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.198177 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.198256 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.198529 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.198599 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 06:23:20.199000 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.190691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43048)
0928 06:23:20.190874 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.190883 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.190943 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.191096 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.191096 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.191411 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.191516 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.191518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.193340 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.193341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.194272 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.194275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.194732 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.194733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.194804 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.195284 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.195289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.195723 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.197462 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.197464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.197464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.197668 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.197880 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.197880 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.197880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.197938 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.198247 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.198248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.198309 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.198781 (+ 472us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.198908 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":2,"spinlock_wait_cycles":10112,"thread_start_us":106,"threads_started":1}
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.126@KRBTEST.COM for oryx/127.31.214.125@KRBTEST.COM
I20260928 06:23:20.209688 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.202046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.31.214.125:34981)
0928 06:23:20.202111 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.202113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.202302 (+ 189us) server_negotiation.cc:407] Connection header received
0928 06:23:20.202385 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.202385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.202394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.202439 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.202935 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.203505 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.204682 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.204858 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.204916 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.207094 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.207095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.207096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.207103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.208479 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.208875 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.208876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.208877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.208935 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.209133 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.209134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.209135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.209243 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.209315 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.209500 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.209578 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:23:20.209697 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.202021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:34981 (local address 127.0.0.1:52116)
0928 06:23:20.202179 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.202231 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.202309 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.202444 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.202448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.202806 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.202869 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.202871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.203580 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.203581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.204580 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.204583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.204934 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.204935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.205032 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.205495 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.205500 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.207007 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.208578 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.208587 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.208589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.208800 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.209002 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.209003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.209004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.209070 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.209321 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.209329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.209380 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.209474 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.209590 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260928 06:23:20.210294 6231 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.214.125:34981: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b0014cf7b5634653bacf2358884671a6"
member_type: NON_VOTER
last_known_addr {
host: "127.31.214.125"
port: 34981
}
attrs {
promote: true
}
}
dest_uuid: "8d549ccb5ce04c039d00ed910205c763"
cas_config_opid_index: -1
I20260928 06:23:20.210635 6231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } }
I20260928 06:23:20.211221 6791 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b0014cf7b5634653bacf2358884671a6 (127.31.214.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } } }
I20260928 06:23:20.211668 6737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:20.212281 6791 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.214.125:34981
I20260928 06:23:20.212513 6737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Refusing update from remote peer 8d549ccb5ce04c039d00ed910205c763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 06:23:20.212563 6695 master.cc:537] Master@127.31.214.125:34981 shutting down...
I20260928 06:23:20.212551 6791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8d549ccb5ce04c039d00ed910205c763" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } } }
I20260928 06:23:20.212623 6791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:20.213243 6790 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:23:20.214094 6216 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 06:23:20.214879 6216 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 -> Peer b0014cf7b5634653bacf2358884671a6 (127.31.214.125:34981): Couldn't send request to peer b0014cf7b5634653bacf2358884671a6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:23:20.216298 6695 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:20.216419 6695 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:20.216480 6695 tablet_replica.cc:330] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: stopping tablet replica
I20260928 06:23:20.219224 6695 master.cc:552] Master@127.31.214.125:34981 shutdown complete.
I20260928 06:23:20.219275 6695 master_runner.cc:305] Clearing existing system tablet
I20260928 06:23:20.219411 6695 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:20.219461 6695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:20.220181 6695 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260928 06:23:20.220764 6795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.220904 6695 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:20.220965 6695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.220990 6695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal
uuid: "b0014cf7b5634653bacf2358884671a6"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "729bccd30b8e342ab74cfca128f094c1"
server_key_iv: "4dbe4e351d140defae41c2aa357284b2"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.221751 6695 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:20.223541 6695 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:23:20.223618 6695 log.cc:1196] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 06:23:20.223822 6695 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: Deleting consensus metadata
I20260928 06:23:20.223948 6695 master_runner.cc:315] Copying system tablet from 127.31.214.126:46093
I20260928 06:23:20.224561 6695 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.214.126:46093
I20260928 06:23:20.231611 6801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.224854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43062)
0928 06:23:20.224989 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.225000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.225073 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.225216 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.225217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.225415 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.225477 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.225479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.226225 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.226226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.227115 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.227118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.227671 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.227672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.227747 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.228180 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.228185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.228590 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.230438 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.230439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.230439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.230726 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.230945 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.230946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.230947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.231015 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.231258 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.231259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.231310 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.231381 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.231492 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":91,"threads_started":1}
I20260928 06:23:20.231623 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.224960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.31.214.126:46093)
0928 06:23:20.225011 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.225013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.225056 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:23:20.225147 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.225147 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.225157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.225208 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.225555 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.226149 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.227260 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.227572 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.227653 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.228685 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.228687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.228687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.228695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.230357 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.230818 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.230819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.230820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.230877 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.231077 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.231078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.231078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.231196 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.231247 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.231481 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.231556 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:23:20.232286 6251 tablet_copy_service.cc:140] P 8d549ccb5ce04c039d00ed910205c763: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b0014cf7b5634653bacf2358884671a6 ({username='oryx', principal='oryx/127.31.214.125@KRBTEST.COM'} at 127.0.0.1:43062)
I20260928 06:23:20.232404 6251 tablet_copy_service.cc:161] P 8d549ccb5ce04c039d00ed910205c763: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b0014cf7b5634653bacf2358884671a6 at {username='oryx', principal='oryx/127.31.214.125@KRBTEST.COM'} at 127.0.0.1:43062: session id = b0014cf7b5634653bacf2358884671a6-00000000000000000000000000000000
I20260928 06:23:20.233075 6251 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:23:20.233780 6695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:23:20.235174 6695 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: tablet copy: Starting download of 0 data blocks...
I20260928 06:23:20.235337 6695 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: tablet copy: Starting download of 1 WAL segments...
I20260928 06:23:20.236387 6695 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:23:20.237404 6251 tablet_copy_service.cc:342] P 8d549ccb5ce04c039d00ed910205c763: Request end of tablet copy session b0014cf7b5634653bacf2358884671a6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.214.125@KRBTEST.COM'} at 127.0.0.1:43062
I20260928 06:23:20.237468 6251 tablet_copy_service.cc:433] P 8d549ccb5ce04c039d00ed910205c763: ending tablet copy session b0014cf7b5634653bacf2358884671a6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b0014cf7b5634653bacf2358884671a6
I20260928 06:23:20.240197 6695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:20.241583 6802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.241592 6803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.243021 6805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.243045 6695 server_base.cc:1034] running on GCE node
I20260928 06:23:20.243261 6695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:20.243422 6695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:20.244544 6695 hybrid_clock.cc:648] HybridClock initialized: now 1790576600244533 us; error 27 us; skew 500 ppm
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:20.246124 6695 init.cc:377] Logged in from keytab as oryx/127.31.214.125@KRBTEST.COM (short username oryx)
I20260928 06:23:20.246546 6695 webserver.cc:460] Webserver started at http://127.31.214.125:38031/ using document root <none> and password file <none>
I20260928 06:23:20.246712 6695 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:20.246745 6695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:20.247454 6695 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:23:20.248085 6812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.248225 6695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:23:20.248278 6695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal
uuid: "b0014cf7b5634653bacf2358884671a6"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "729bccd30b8e342ab74cfca128f094c1"
server_key_iv: "4dbe4e351d140defae41c2aa357284b2"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.248354 6695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:20.253247 6695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:20.254972 6695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:20.258323 6695 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.125:34981
I20260928 06:23:20.258662 6864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.125:34981 every 8 connection(s)
I20260928 06:23:20.258931 6695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
I20260928 06:23:20.259829 6865 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:23:20.260337 6865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: Bootstrap starting.
I20260928 06:23:20.263276 6865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:23:20.263540 6865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: Bootstrap complete.
I20260928 06:23:20.263752 6865 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } }
I20260928 06:23:20.263839 6865 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b0014cf7b5634653bacf2358884671a6, State: Initialized, Role: LEARNER
I20260928 06:23:20.263871 6865 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } }
I20260928 06:23:20.264055 6865 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:20.264063 6868 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } } }
I20260928 06:23:20.264209 6868 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:23:20.265579 6879 catalog_manager.cc:1259] Loaded cluster ID: 2a770ff9ca0e4473b854b1176c2566bb
I20260928 06:23:20.265632 6879 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: loading cluster ID for follower catalog manager: success
I20260928 06:23:20.266757 6879 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: acquiring CA information for follower catalog manager: success
I20260928 06:23:20.267078 6879 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:23:20.285710 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.278272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.31.214.125:34981)
0928 06:23:20.278485 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.278487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.278499 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:23:20.278529 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.278530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.278543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.278625 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.279219 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.280255 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.280991 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.281290 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.281376 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.282749 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.282751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.282751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.282759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.284430 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.284823 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.284824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.284825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.284893 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.285101 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.285102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.285103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.285248 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.285356 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.285491 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.285580 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for oryx/127.31.214.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.124.keytab.
Entry for principal oryx/127.31.214.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.124.keytab.
I20260928 06:23:20.324843 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.124:38083
--webserver_interface=127.31.214.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.124.keytab
--principal=oryx/127.31.214.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:20.410039 6883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:20.410239 6883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:20.410261 6883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:20.411445 6883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:20.411489 6883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:20.411502 6883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:20.411515 6883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:20.411557 6883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:20.413062 6883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.124.keytab
--principal=oryx/127.31.214.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.124:38083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.214.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:20.413322 6883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:20.413568 6883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:20.418821 6889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.418830 6891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.419130 6883 server_base.cc:1034] running on GCE node
W20260928 06:23:20.418840 6888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.419492 6883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:20.419806 6883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:20.420974 6883 hybrid_clock.cc:648] HybridClock initialized: now 1790576600420960 us; error 28 us; skew 500 ppm
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:20.423637 6883 init.cc:377] Logged in from keytab as oryx/127.31.214.124@KRBTEST.COM (short username oryx)
I20260928 06:23:20.424171 6883 webserver.cc:460] Webserver started at http://127.31.214.124:33971/ using document root <none> and password file <none>
I20260928 06:23:20.424428 6883 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:20.424475 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:20.424638 6883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:20.425758 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data/instance:
uuid: "9bc67401c85c4bb88ae6abd1824f6cc2"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "0f789845643dbf996bf3e18004dbefe1"
server_key_iv: "771046b050df31134d54c1ab5df2d6d4"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.426160 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal/instance:
uuid: "9bc67401c85c4bb88ae6abd1824f6cc2"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "0f789845643dbf996bf3e18004dbefe1"
server_key_iv: "771046b050df31134d54c1ab5df2d6d4"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.427575 6883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:20.428366 6898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.428670 6883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:23:20.428804 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal
uuid: "9bc67401c85c4bb88ae6abd1824f6cc2"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "0f789845643dbf996bf3e18004dbefe1"
server_key_iv: "771046b050df31134d54c1ab5df2d6d4"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.428872 6883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:20.455329 6883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:20.457533 6883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:20.457693 6883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:20.462158 6883 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.124:38083
I20260928 06:23:20.462211 6950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.124:38083 every 8 connection(s)
I20260928 06:23:20.462604 6883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
I20260928 06:23:20.463647 6951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:20.465389 6951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:23:20.470391 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 6883
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.124@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:20.475122 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.466292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.31.214.126:46093)
0928 06:23:20.466376 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.466384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.466547 (+ 163us) server_negotiation.cc:407] Connection header received
0928 06:23:20.466989 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.466991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.467005 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.467076 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.467680 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.468173 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.469306 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.469505 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.469571 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.472030 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.472031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.472031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.472040 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.473478 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.473937 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.473938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.473938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.474050 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.474353 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.474354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.474355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.474494 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.474561 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.474922 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.475010 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:23:20.475333 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.466208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43070)
0928 06:23:20.466391 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.466434 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.466890 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.467134 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.467137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.467398 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.467600 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.467603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.468286 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.468287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.469142 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.469144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.469598 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.469599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.469717 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.470349 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.470361 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.471957 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.473559 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.473566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.473568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.473854 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.474117 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.474118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.474119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.474225 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.474584 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.474591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.474764 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.474891 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.475052 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260928 06:23:20.475971 6951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.214.124@KRBTEST.COM
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.124@KRBTEST.COM for oryx/127.31.214.125@KRBTEST.COM
I20260928 06:23:20.480921 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.470990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.31.214.124:38083)
0928 06:23:20.471171 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.471176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.471193 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:23:20.471211 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.471212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.471228 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.471335 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.471971 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.473083 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.474020 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.474464 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.474542 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.477341 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.477348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.477348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.477357 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.479149 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.480067 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.480070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.480071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.480123 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.480349 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.480350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.480350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.480494 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.480659 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.480760 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.480847 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260928 06:23:20.482898 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.476413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:34981 (local address 127.0.0.1:52140)
0928 06:23:20.476469 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.476481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.476521 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.476693 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.476694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.476833 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.476880 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.476881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.477555 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.477556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.478145 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.478147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.478527 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.478528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.478607 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.479019 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.479024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.480231 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.482109 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.482110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.482110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.482266 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.482476 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.482476 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.482476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.482522 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.482687 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.482687 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.482718 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.482761 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.482838 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:23:20.482951 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.476504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.31.214.125:34981)
0928 06:23:20.476580 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.476582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.476589 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:23:20.476617 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.476618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.476628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.476689 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.476969 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.477454 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.478247 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.478454 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.478531 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.480337 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.480338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.480338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.480346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.482035 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.482384 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.482385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.482385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.482435 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.482555 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.482555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.482555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.482645 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.482693 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.482776 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.482870 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:23:20.483569 6951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38083 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.124@KRBTEST.COM for oryx/127.31.214.124@KRBTEST.COM
I20260928 06:23:20.489719 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.483817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38083 (local address 127.0.0.1:35894)
0928 06:23:20.483868 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.483880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.483923 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.484038 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.484038 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.484144 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.484174 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.484175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.484786 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.484787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.485244 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.485245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.485603 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.485604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.485643 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.486072 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.486087 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.487341 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.488906 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.488907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.488907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.489060 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.489235 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.489235 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.489236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.489285 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.489493 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.489494 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.489528 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.489579 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.489654 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:23:20.489879 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.483886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.31.214.124:38083)
0928 06:23:20.483939 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.483940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.483948 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:23:20.483972 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.483973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.483985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.484028 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.484230 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.484715 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.485351 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.485537 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.485591 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.487418 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.487419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.487420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.487426 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.488828 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.489128 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.489129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.489129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.489175 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.489345 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.489345 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.489345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.489438 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.489486 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.489593 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.489741 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":34}
I20260928 06:23:20.491222 6951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: Bootstrap starting.
I20260928 06:23:20.492043 6951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:20.492295 6951 log.cc:824] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:20.493013 6951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: No bootstrap required, opened a new log
I20260928 06:23:20.494578 6951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38083 } }
I20260928 06:23:20.494704 6951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:20.494727 6951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc67401c85c4bb88ae6abd1824f6cc2, State: Initialized, Role: FOLLOWER
I20260928 06:23:20.494854 6951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38083 } }
I20260928 06:23:20.495136 6951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:20.495131 6956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38083 } } }
I20260928 06:23:20.495249 6956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:23:20.496969 6967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:23:20.497035 6967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:23:20.499711 6883 master_runner.cc:428] Detected that this master 9bc67401c85c4bb88ae6abd1824f6cc2 is joining an existing cluster
I20260928 06:23:20.499758 6883 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.214.124:38083
I20260928 06:23:20.533576 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 6293
I20260928 06:23:20.535055 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.500801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:34981 (local address 127.0.0.1:52146)
0928 06:23:20.500945 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.500957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.501017 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.501202 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.501203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.501425 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.501514 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.501516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.502593 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.502594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.503609 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.503612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.504212 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.504213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.504300 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.504816 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.504821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.530814 (+ 25993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.533462 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.533463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.533463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.533689 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.534296 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.534297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.534297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.534355 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.534668 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.534669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.534722 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.534788 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.534932 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260928 06:23:20.535073 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.501091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.31.214.126:46093)
0928 06:23:20.501139 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.501142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.501725 (+ 583us) server_negotiation.cc:407] Connection header received
0928 06:23:20.501836 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.501838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.501849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.501880 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.502353 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.503135 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.504264 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.504566 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.504612 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.531525 (+ 26913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.531527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.531527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.531537 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.533850 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.534269 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.534271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.534272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.534345 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.534546 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.534547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.534547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.534679 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.534729 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.534909 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.534985 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:23:20.535436 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.501155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43086)
0928 06:23:20.501670 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.501679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.501750 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.501966 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.501976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.502185 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.502256 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.502258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.503261 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.503262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.504157 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.504159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.504698 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.504699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.504769 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.505200 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.505205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.531407 (+ 26202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.533929 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.533930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.533931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.534175 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.534423 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.534424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.534424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.534476 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.534761 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.534762 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.534798 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.535186 (+ 388us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.535334 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":124,"threads_started":1}
I20260928 06:23:20.535444 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.500887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.31.214.125:34981)
0928 06:23:20.500935 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.500937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.501003 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:23:20.501073 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.501074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.501084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.501147 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.501642 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.502495 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.503753 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.504082 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.504142 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.530978 (+ 26836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.530980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.530980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.530990 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.533358 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.534159 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.534161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.534161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.534220 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.534427 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.534428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.534429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.534583 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.534642 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.535277 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.535356 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:23:20.538976 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.501412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.31.214.124:38083)
0928 06:23:20.501446 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.501448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.506238 (+ 4790us) server_negotiation.cc:407] Connection header received
0928 06:23:20.506285 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.506286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.506297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.506351 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.506717 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.507435 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.508518 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.508802 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.508876 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.531115 (+ 22239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.531117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.531117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.531124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.533348 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.538124 (+ 4776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.538125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.538125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.538177 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.538357 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.538358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.538358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.538504 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.538557 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.538741 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.538822 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:23:20.539353 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.501469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38083 (local address 127.0.0.1:35902)
0928 06:23:20.506148 (+ 4679us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.506168 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.506231 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.506354 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.506355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.506569 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.506638 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.506640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.507526 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.507527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.508427 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.508429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.508884 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.508885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.508936 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.509410 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.509415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.530994 (+ 21579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.537754 (+ 6760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.537754 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.537755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.538041 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.538238 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.538238 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.538238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.538290 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.538584 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.538585 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.538636 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.539087 (+ 451us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.539195 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4653,"thread_start_us":74,"threads_started":1}
I20260928 06:23:20.540879 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:36869
--local_ip_for_outbound_sockets=127.31.214.65
--tserver_master_addrs=127.31.214.126:46093
--webserver_port=45091
--webserver_interface=127.31.214.65
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab
--principal=oryx/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.214.126:46093,127.31.214.125:34981
--tserver_master_addrs=127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576599, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.126@KRBTEST.COM for oryx/127.31.214.124@KRBTEST.COM
I20260928 06:23:20.546590 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.537640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38083 (local address 127.0.0.1:35910)
0928 06:23:20.537690 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.537702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.537749 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.538222 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.538224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.538392 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.538447 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.538449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.540085 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.540087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.540833 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.540837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.541361 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.541362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.541444 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.541901 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.541906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.543537 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.545472 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.545473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.545473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.545653 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.545868 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.545869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.545869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.545929 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.546227 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.546228 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.546276 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.546367 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.546486 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:23:20.546600 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.537734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.31.214.124:38083)
0928 06:23:20.538064 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.538066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.538074 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:23:20.538113 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.538114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.538125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.538516 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.538557 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.539984 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.540939 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.541275 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.541362 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.543653 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.543655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.543656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.543662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.545369 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.545729 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.545730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.545731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.545797 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.546028 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.546030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.546030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.546154 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.546235 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.546406 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.546485 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":104,"threads_started":1}
I20260928 06:23:20.547509 6231 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.214.124:38083: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2"
member_type: NON_VOTER
last_known_addr {
host: "127.31.214.124"
port: 38083
}
attrs {
promote: true
}
}
dest_uuid: "8d549ccb5ce04c039d00ed910205c763"
cas_config_opid_index: 5
I20260928 06:23:20.547812 6231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:20.548213 6790 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9bc67401c85c4bb88ae6abd1824f6cc2 (127.31.214.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.548569 6925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:20.549414 6790 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.214.124:38083
I20260928 06:23:20.549494 6790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8d549ccb5ce04c039d00ed910205c763" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.549551 6790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:20.550055 6925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [term 1 FOLLOWER]: Refusing update from remote peer 8d549ccb5ce04c039d00ed910205c763: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:23:20.550318 6883 master.cc:537] Master@127.31.214.124:38083 shutting down...
I20260928 06:23:20.550504 6791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:23:20.551203 6217 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 -> Peer 9bc67401c85c4bb88ae6abd1824f6cc2 (127.31.214.124:38083): Couldn't send request to peer 9bc67401c85c4bb88ae6abd1824f6cc2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:23:20.555104 6883 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:20.555228 6883 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:20.555228 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.550120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.31.214.125:34981)
0928 06:23:20.550219 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.550221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.550230 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:23:20.550255 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.550256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.550280 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.550346 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:20.551441 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.552158 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.554130 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.554627 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.554669 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.554721 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.555105 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.555153 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 06:23:20.555280 6883 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: stopping tablet replica
I20260928 06:23:20.556051 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.549772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:34981 (local address 127.0.0.1:52160)
0928 06:23:20.549831 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.549841 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.549882 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.551288 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.551289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.551291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:20.551356 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.551358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.552867 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.552868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.553886 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.553889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.554912 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.554913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.554990 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.555015 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.555336 (+ 321us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.555382 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:23:20.556411 6839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 LEARNER]: Refusing update from remote peer 8d549ccb5ce04c039d00ed910205c763: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:23:20.556799 6978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 06:23:20.557406 6839 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9bc67401c85c4bb88ae6abd1824f6cc2 (127.31.214.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.558583 6980 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d549ccb5ce04c039d00ed910205c763" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.558743 6980 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:23:20.558826 6868 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d549ccb5ce04c039d00ed910205c763. Latest consensus state: current_term: 1 leader_uuid: "8d549ccb5ce04c039d00ed910205c763" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: NON_VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: true } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.558929 6868 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:23:20.559479 6978 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763: attempting to promote NON_VOTER b0014cf7b5634653bacf2358884671a6 to VOTER
I20260928 06:23:20.559679 6978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:20.560186 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.560042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38083 (local address 127.0.0.1:35916)
0928 06:23:20.560101 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.560152 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:23:20.560293 6839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 LEARNER]: Refusing update from remote peer 8d549ccb5ce04c039d00ed910205c763: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 06:23:20.560525 6791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 06:23:20.560711 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.560580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38083 (local address 127.0.0.1:35922)
0928 06:23:20.560660 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.560688 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 06:23:20.560828 6217 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 -> Peer 9bc67401c85c4bb88ae6abd1824f6cc2 (127.31.214.124:38083): Couldn't send request to peer 9bc67401c85c4bb88ae6abd1824f6cc2. Status: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:23:20.561131 6791 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b0014cf7b5634653bacf2358884671a6 (127.31.214.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.561276 6839 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b0014cf7b5634653bacf2358884671a6 (127.31.214.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.562146 6791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8d549ccb5ce04c039d00ed910205c763" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.562179 6790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d549ccb5ce04c039d00ed910205c763" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.562222 6791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:20.562371 6868 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d549ccb5ce04c039d00ed910205c763" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.562438 6868 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:23:20.562230 6790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:20.569069 6883 master.cc:552] Master@127.31.214.124:38083 shutdown complete.
I20260928 06:23:20.569136 6883 master_runner.cc:305] Clearing existing system tablet
I20260928 06:23:20.569273 6883 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:20.569391 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:20.570091 6883 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:20.570612 6983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.570768 6883 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:20.570830 6883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.570852 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal
uuid: "9bc67401c85c4bb88ae6abd1824f6cc2"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "0f789845643dbf996bf3e18004dbefe1"
server_key_iv: "771046b050df31134d54c1ab5df2d6d4"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.571429 6883 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:20.572695 6883 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:23:20.572778 6883 log.cc:1196] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 06:23:20.572986 6883 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: Deleting consensus metadata
I20260928 06:23:20.573129 6883 master_runner.cc:315] Copying system tablet from 127.31.214.126:46093
I20260928 06:23:20.573786 6883 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: tablet copy: Beginning tablet copy session from remote peer at address 127.31.214.126:46093
I20260928 06:23:20.580188 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.574144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43102)
0928 06:23:20.574296 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.574307 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.574396 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.574539 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.574539 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.574718 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.574768 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.574770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.575465 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.575467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.576110 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.576112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.576506 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.576508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.576584 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.577088 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.577093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.577405 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.579117 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.579118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.579118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.579305 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.579516 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.579517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.579517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.579581 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.579841 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.579842 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.579892 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.579974 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.580086 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 06:23:20.580188 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.574195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.31.214.126:46093)
0928 06:23:20.574242 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.574244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.574380 (+ 136us) server_negotiation.cc:407] Connection header received
0928 06:23:20.574462 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.574463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.574475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.574534 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.574859 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.575373 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.576197 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.576419 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.576495 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.577503 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.577504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.577505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.577512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.579010 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.579379 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.579381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.579382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.579443 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.579652 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.579653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.579654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.579768 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.579850 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.580018 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.580095 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:23:20.580762 6251 tablet_copy_service.cc:140] P 8d549ccb5ce04c039d00ed910205c763: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9bc67401c85c4bb88ae6abd1824f6cc2 ({username='oryx', principal='oryx/127.31.214.124@KRBTEST.COM'} at 127.0.0.1:43102)
I20260928 06:23:20.580837 6251 tablet_copy_service.cc:161] P 8d549ccb5ce04c039d00ed910205c763: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9bc67401c85c4bb88ae6abd1824f6cc2 at {username='oryx', principal='oryx/127.31.214.124@KRBTEST.COM'} at 127.0.0.1:43102: session id = 9bc67401c85c4bb88ae6abd1824f6cc2-00000000000000000000000000000000
I20260928 06:23:20.581329 6251 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:23:20.581796 6883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:23:20.582864 6883 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: tablet copy: Starting download of 0 data blocks...
I20260928 06:23:20.583021 6883 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: tablet copy: Starting download of 1 WAL segments...
I20260928 06:23:20.583730 6883 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:23:20.584460 6251 tablet_copy_service.cc:342] P 8d549ccb5ce04c039d00ed910205c763: Request end of tablet copy session 9bc67401c85c4bb88ae6abd1824f6cc2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.214.124@KRBTEST.COM'} at 127.0.0.1:43102
I20260928 06:23:20.584514 6251 tablet_copy_service.cc:433] P 8d549ccb5ce04c039d00ed910205c763: ending tablet copy session 9bc67401c85c4bb88ae6abd1824f6cc2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9bc67401c85c4bb88ae6abd1824f6cc2
I20260928 06:23:20.587044 6883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:20.588392 6990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.588550 6993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.588733 6991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.588764 6883 server_base.cc:1034] running on GCE node
I20260928 06:23:20.588936 6883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:20.589119 6883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:20.590231 6883 hybrid_clock.cc:648] HybridClock initialized: now 1790576600590223 us; error 28 us; skew 500 ppm
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:20.591625 6883 init.cc:377] Logged in from keytab as oryx/127.31.214.124@KRBTEST.COM (short username oryx)
I20260928 06:23:20.591933 6883 webserver.cc:460] Webserver started at http://127.31.214.124:45605/ using document root <none> and password file <none>
I20260928 06:23:20.592057 6883 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:20.592093 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:20.592753 6883 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:23:20.593304 7000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.593441 6883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:23:20.593500 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal
uuid: "9bc67401c85c4bb88ae6abd1824f6cc2"
format_stamp: "Formatted at 2026-09-28 06:23:20 on dist-test-slave-q03f"
server_key: "0f789845643dbf996bf3e18004dbefe1"
server_key_iv: "771046b050df31134d54c1ab5df2d6d4"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.593568 6883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:20.598815 6883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:20.600721 6883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:20.604182 6883 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.124:38083
I20260928 06:23:20.604606 6883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
I20260928 06:23:20.605547 7053 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:23:20.606060 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: Bootstrap starting.
I20260928 06:23:20.606069 7052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.124:38083 every 8 connection(s)
I20260928 06:23:20.608423 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:23:20.608670 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: Bootstrap complete.
I20260928 06:23:20.608870 7053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:20.608961 7053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9bc67401c85c4bb88ae6abd1824f6cc2, State: Initialized, Role: LEARNER
I20260928 06:23:20.609005 7053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:20.609208 7053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:20.609217 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } } }
I20260928 06:23:20.609445 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:23:20.610777 7067 catalog_manager.cc:1259] Loaded cluster ID: 2a770ff9ca0e4473b854b1176c2566bb
I20260928 06:23:20.610879 7067 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: loading cluster ID for follower catalog manager: success
I20260928 06:23:20.612057 7067 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: acquiring CA information for follower catalog manager: success
I20260928 06:23:20.612349 7067 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9bc67401c85c4bb88ae6abd1824f6cc2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:23:20.662501 6977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:20.662751 6977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:20.662801 6977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:20.664085 6977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:20.664162 6977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:20.664211 6977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:20.665910 6977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab
--principal=oryx/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:36869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=45091
--tserver_master_addrs=127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:20.666258 6977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:20.666517 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:20.672827 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.672833 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.673079 6977 server_base.cc:1034] running on GCE node
W20260928 06:23:20.673213 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.673496 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:20.673847 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:20.675032 6977 hybrid_clock.cc:648] HybridClock initialized: now 1790576600675031 us; error 52 us; skew 500 ppm
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:20.678218 6977 init.cc:377] Logged in from keytab as oryx/127.31.214.65@KRBTEST.COM (short username oryx)
I20260928 06:23:20.678746 6977 webserver.cc:460] Webserver started at http://127.31.214.65:45091/ using document root <none> and password file <none>
I20260928 06:23:20.678999 6977 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:20.679080 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:20.680492 6977 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:20.681104 7082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.681281 6977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:20.681396 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "edcac2b2bcd1470ba1bd11588faec84b"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "c068212924441fe9a4fc4e6600ee5e3a"
server_key_iv: "cb85d0f97f5e6ed56270e634236b49c4"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.681761 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:20.694409 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:20.696379 6977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:20.696544 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:20.696779 6977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:20.697096 6977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:20.697149 6977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.697191 6977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:20.697218 6977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.703367 6977 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:36869
I20260928 06:23:20.703428 7195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:36869 every 8 connection(s)
I20260928 06:23:20.703898 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:23:20.707682 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 6977
I20260928 06:23:20.707847 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 6427
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.125@KRBTEST.COM
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.124@KRBTEST.COM
I20260928 06:23:20.716439 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:43369
--local_ip_for_outbound_sockets=127.31.214.66
--tserver_master_addrs=127.31.214.126:46093
--webserver_port=42077
--webserver_interface=127.31.214.66
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab
--principal=oryx/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.214.126:46093,127.31.214.125:34981
--tserver_master_addrs=127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:23:20.718359 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.704692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:40547 (local address 127.31.214.126:46093)
0928 06:23:20.704750 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.704753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.706139 (+ 1386us) server_negotiation.cc:407] Connection header received
0928 06:23:20.706457 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.706458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.706473 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.706506 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.708528 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.709151 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.710749 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.711125 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.711196 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.714426 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.714427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.714428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.714437 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.716217 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.717171 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.717172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.717173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.717232 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.717541 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.717542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.717542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.717699 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.717756 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.718162 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.718253 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:23:20.718886 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.705060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:53649 (local address 127.31.214.124:38083)
0928 06:23:20.705207 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.705210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.706266 (+ 1056us) server_negotiation.cc:407] Connection header received
0928 06:23:20.706516 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.706518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.706530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.706604 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.709273 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.711262 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.712170 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.712515 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.712583 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.715622 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.715623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.715624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.715631 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.717422 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.717905 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.717906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.717907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.717994 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.718204 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.718204 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.718205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.718345 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.718406 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.718646 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.718745 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260928 06:23:20.718932 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.704662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:40547)
0928 06:23:20.706061 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.706071 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.706360 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.706598 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.706603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.707018 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.707250 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.707255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.709265 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.709266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.710656 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.710659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.711265 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.711266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.711421 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.712043 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.712054 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.714334 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.716306 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.716315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.716318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.717069 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.717318 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.717318 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.717319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.717458 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.717790 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.717797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.717998 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.718454 (+ 456us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.718668 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"mutex_wait_us":58,"thread_start_us":98,"threads_started":1}
I20260928 06:23:20.719102 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.704748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:52637 (local address 127.31.214.125:34981)
0928 06:23:20.704797 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.704799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.705225 (+ 426us) server_negotiation.cc:407] Connection header received
0928 06:23:20.706448 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.706449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.706463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.706518 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.707360 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.708040 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.711110 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.711411 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.711469 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.715272 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.715273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.715273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.715282 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.717546 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.717984 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.717985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.717986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.718045 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.718250 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.718251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.718251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.718388 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.718443 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.718931 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.719016 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:23:20.719187 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.704969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38083 (local address 127.31.214.65:53649)
0928 06:23:20.706184 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.706192 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.706359 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.706618 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.706619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.707031 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.707251 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.707257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.711341 (+ 4084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.711342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.712033 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.712036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.712639 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.712640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.712710 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.713122 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.713126 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.715530 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.717515 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.717516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.717517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.717816 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.718074 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.718075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.718075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.718137 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.718432 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.718433 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.718486 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.718551 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.719064 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"mutex_wait_us":404,"thread_start_us":57,"threads_started":1}
I20260928 06:23:20.719462 7197 heartbeater.cc:347] Connected to a master server at 127.31.214.126:46093
I20260928 06:23:20.719552 7197 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:20.719753 7197 heartbeater.cc:511] Master 127.31.214.126:46093 requested a full tablet report, sending...
I20260928 06:23:20.720063 7196 heartbeater.cc:347] Connected to a master server at 127.31.214.124:38083
I20260928 06:23:20.720141 7196 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:20.720216 7196 heartbeater.cc:511] Master 127.31.214.124:38083 requested a full tablet report, sending...
I20260928 06:23:20.720224 6231 ts_manager.cc:194] Re-registered known tserver with Master: edcac2b2bcd1470ba1bd11588faec84b (127.31.214.65:36869)
I20260928 06:23:20.720636 6231 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:40547
I20260928 06:23:20.720649 7017 ts_manager.cc:194] Registered new tserver with Master: edcac2b2bcd1470ba1bd11588faec84b (127.31.214.65:36869)
I20260928 06:23:20.720929 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.704662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:34981 (local address 127.31.214.65:52637)
0928 06:23:20.705059 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.705109 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.706341 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.706598 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.706602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.707022 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.707251 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.707257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.709397 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.709398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.711025 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.711027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.711522 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.711522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.711584 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.712209 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.712214 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.715194 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.717627 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.717628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.717628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.717880 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.718116 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.718117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.718117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.718176 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.718733 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.718734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.718774 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.719895 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.720022 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":946,"thread_start_us":107,"threads_started":1}
I20260928 06:23:20.722123 7198 heartbeater.cc:347] Connected to a master server at 127.31.214.125:34981
I20260928 06:23:20.722189 7198 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:20.722266 7198 heartbeater.cc:511] Master 127.31.214.125:34981 requested a full tablet report, sending...
I20260928 06:23:20.722766 6829 ts_manager.cc:194] Registered new tserver with Master: edcac2b2bcd1470ba1bd11588faec84b (127.31.214.65:36869)
W20260928 06:23:20.805758 7204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:20.806061 7204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:20.806090 7204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:20.807229 7204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:20.807282 7204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:20.807309 7204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:20.808751 7204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab
--principal=oryx/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:43369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=42077
--tserver_master_addrs=127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:20.808964 7204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:20.809271 7204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:20.814770 7212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.814779 7209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.814779 7210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.814837 7204 server_base.cc:1034] running on GCE node
I20260928 06:23:20.815402 7204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:20.815610 7204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:20.816846 7204 hybrid_clock.cc:648] HybridClock initialized: now 1790576600816819 us; error 37 us; skew 500 ppm
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:20.819631 7204 init.cc:377] Logged in from keytab as oryx/127.31.214.66@KRBTEST.COM (short username oryx)
I20260928 06:23:20.820107 7204 webserver.cc:460] Webserver started at http://127.31.214.66:42077/ using document root <none> and password file <none>
I20260928 06:23:20.820374 7204 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:20.820464 7204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:20.821846 7204 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:23:20.822515 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.822690 7204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:23:20.822788 7204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "d8e4de63f2044696abbf32f9687fb04c"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "5b83b16b7e04ecfe0804b2e24b52abb6"
server_key_iv: "fd33a85d3d8b393bbb138bcbf9574ab4"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.823159 7204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:20.834656 7204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:20.837085 7204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:20.837261 7204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:20.837522 7204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:20.837855 7204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:20.837920 7204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.838042 7204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:20.838076 7204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.844139 7204 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:43369
I20260928 06:23:20.844182 7332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:43369 every 8 connection(s)
I20260928 06:23:20.844552 7204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.124@KRBTEST.COM
I20260928 06:23:20.852902 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 7204
I20260928 06:23:20.853070 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 6561
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.125@KRBTEST.COM
I20260928 06:23:20.856846 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.845370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:40867 (local address 127.31.214.126:46093)
0928 06:23:20.845581 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.845582 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.845901 (+ 319us) server_negotiation.cc:407] Connection header received
0928 06:23:20.847042 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.847043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.847056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.847111 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.847787 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.848321 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.849836 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.850165 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.850240 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.852858 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.852860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.852860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.852874 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.854732 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.855682 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.855684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.855685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.855747 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.856068 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.856069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.856070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.856222 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.856300 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.856643 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.856731 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260928 06:23:20.857110 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.845363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:40867)
0928 06:23:20.845827 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.845835 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.846723 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.847131 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.847132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.847474 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.847660 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.847666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.848523 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.848524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.849746 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.849748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.850250 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.850251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.850399 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.851046 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.851057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.852746 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.855239 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.855246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.855249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.855557 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.855829 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.855830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.855831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.855987 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.856303 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.856310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.856484 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.856604 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.856812 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":138,"thread_start_us":102,"threads_started":1}
I20260928 06:23:20.857728 7334 heartbeater.cc:347] Connected to a master server at 127.31.214.126:46093
I20260928 06:23:20.857828 7334 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:20.857903 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.845719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:37251 (local address 127.31.214.124:38083)
0928 06:23:20.845763 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.845765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.845894 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:23:20.846920 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.846922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.846934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.846981 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.847816 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.848591 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.850405 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.850612 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.850708 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.854674 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.854676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.854676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.854685 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.856363 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.856747 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.856748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.856749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.856804 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.857280 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.857281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.857282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.857414 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.857458 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.857723 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.857815 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:23:20.858057 7334 heartbeater.cc:511] Master 127.31.214.126:46093 requested a full tablet report, sending...
I20260928 06:23:20.858467 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.845645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38083 (local address 127.31.214.66:37251)
0928 06:23:20.845828 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.845837 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.846711 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.847006 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.847007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.847482 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.847723 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.847725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.848672 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.848673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.849809 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.849810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.850713 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.850714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.850757 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.851215 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.851219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.854563 (+ 3344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.856442 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.856443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.856444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.856656 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.856876 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.856876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.856877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.857182 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.857505 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.857506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.857570 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.857684 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.858359 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":84,"thread_start_us":64,"threads_started":1}
I20260928 06:23:20.858520 6231 ts_manager.cc:194] Re-registered known tserver with Master: d8e4de63f2044696abbf32f9687fb04c (127.31.214.66:43369)
I20260928 06:23:20.858822 7333 heartbeater.cc:347] Connected to a master server at 127.31.214.124:38083
I20260928 06:23:20.858901 7333 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:20.858974 7333 heartbeater.cc:511] Master 127.31.214.124:38083 requested a full tablet report, sending...
I20260928 06:23:20.859078 6231 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:40867
I20260928 06:23:20.859268 7017 ts_manager.cc:194] Registered new tserver with Master: d8e4de63f2044696abbf32f9687fb04c (127.31.214.66:43369)
I20260928 06:23:20.861281 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:42441
--local_ip_for_outbound_sockets=127.31.214.67
--tserver_master_addrs=127.31.214.126:46093
--webserver_port=43309
--webserver_interface=127.31.214.67
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab
--principal=oryx/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:36277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.214.126:46093,127.31.214.125:34981
--tserver_master_addrs=127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:23:20.862102 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.845382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:35255 (local address 127.31.214.125:34981)
0928 06:23:20.845434 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.845437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.845660 (+ 223us) server_negotiation.cc:407] Connection header received
0928 06:23:20.846832 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.846833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.846847 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.846917 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.847765 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.848564 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.849970 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.850295 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.850363 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.859456 (+ 9093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:20.859457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:20.859457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:20.859465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:20.860974 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.861286 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.861288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.861288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.861344 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:20.861544 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:20.861545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:20.861545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:20.861657 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:20.861738 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:20.861894 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:23:20.862012 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:23:20.862273 7337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.845363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:34981 (local address 127.31.214.66:35255)
0928 06:23:20.845536 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.845582 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.846708 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.846996 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.847000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.847442 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.847658 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.847663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.848654 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.848655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.849807 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.849810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.850374 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.850374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:20.850427 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:20.851089 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:20.851093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:20.855061 (+ 3968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:20.861038 (+ 5977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.861039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.861039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.861213 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.861427 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:20.861428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:20.861428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:20.861488 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:20.861722 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:20.861723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:20.861764 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:23:20.861869 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:23:20.862003 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":140,"thread_start_us":98,"threads_started":1}
I20260928 06:23:20.862658 7335 heartbeater.cc:347] Connected to a master server at 127.31.214.125:34981
I20260928 06:23:20.862741 7335 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:20.862819 7335 heartbeater.cc:511] Master 127.31.214.125:34981 requested a full tablet report, sending...
I20260928 06:23:20.863134 6829 ts_manager.cc:194] Registered new tserver with Master: d8e4de63f2044696abbf32f9687fb04c (127.31.214.66:43369)
W20260928 06:23:20.954211 7340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:20.954481 7340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:20.954522 7340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:20.955807 7340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:20.955885 7340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:20.955935 7340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:20.957624 7340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:36277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab
--principal=oryx/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:42441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=43309
--tserver_master_addrs=127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:20.957931 7340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:20.958225 7340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:20.963568 7346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.963573 7348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:20.963603 7345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:20.963929 7340 server_base.cc:1034] running on GCE node
I20260928 06:23:20.964250 7340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:20.964579 7340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:20.965787 7340 hybrid_clock.cc:648] HybridClock initialized: now 1790576600965754 us; error 50 us; skew 500 ppm
Sep 28 06:23:20 dist-test-slave-q03f krb5kdc[6186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:20.968778 7340 init.cc:377] Logged in from keytab as oryx/127.31.214.67@KRBTEST.COM (short username oryx)
I20260928 06:23:20.969295 7340 webserver.cc:460] Webserver started at http://127.31.214.67:43309/ using document root <none> and password file <none>
I20260928 06:23:20.969532 7340 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:20.969594 7340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:20.971158 7340 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:20.971822 7355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.972028 7340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:20.972141 7340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "ff725a15eb8f429eb9c4c76a8c46dafb"
format_stamp: "Formatted at 2026-09-28 06:23:19 on dist-test-slave-q03f"
server_key: "a0d1442f33b55a8ffc363af75747cac7"
server_key_iv: "e45aef843a44d06ac11be99bd824b8e5"
server_key_version: "encryptionkey@0"
I20260928 06:23:20.972498 7340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:20.985312 7340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:20.987691 7340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:20.987867 7340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:20.988138 7340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:20.988468 7340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:20.988521 7340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.988562 7340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:20.988591 7340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:20.995376 7340 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:42441
I20260928 06:23:20.995420 7468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:42441 every 8 connection(s)
I20260928 06:23:20.995774 7340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:20.997009 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 7340
Sep 28 06:23:21 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
Sep 28 06:23:21 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.124@KRBTEST.COM
Sep 28 06:23:21 dist-test-slave-q03f krb5kdc[6186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576600, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.125@KRBTEST.COM
I20260928 06:23:21.009747 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.996558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:54233 (local address 127.31.214.124:38083)
0928 06:23:20.996643 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.996646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.996861 (+ 215us) server_negotiation.cc:407] Connection header received
0928 06:23:20.997601 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.997603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.997617 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.997702 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.998757 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.999580 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.001175 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.001490 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.001532 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.006276 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:21.006278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:21.006278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:21.006285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:21.008002 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.008502 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.008503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.008504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.008581 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.008912 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.008913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.008914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.009046 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:21.009122 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.009550 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.009649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:23:21.010177 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.996564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38083 (local address 127.31.214.67:54233)
0928 06:23:20.996722 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.996764 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.997489 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.997782 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.997786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.998314 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.998645 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.998652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.999699 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.999700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.001076 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.001079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.001625 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.001625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.001772 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.002656 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:21.002666 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:21.006176 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:21.008086 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:21.008093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:21.008096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:21.008417 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:21.008669 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:21.008670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:21.008670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:21.008830 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:21.009192 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:21.009199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:21.009388 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 06:23:21.009638 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 06:23:21.009876 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":112,"thread_start_us":87,"threads_started":1}
I20260928 06:23:21.010336 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.997267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:56163 (local address 127.31.214.125:34981)
0928 06:23:20.997320 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.997323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.997799 (+ 476us) server_negotiation.cc:407] Connection header received
0928 06:23:20.997856 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.997857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.997871 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.997934 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.998812 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.999581 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.001198 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.001493 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.001534 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.006720 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:21.006722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:21.006722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:21.006729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:21.008291 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.008762 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.008763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.008763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.008822 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.009235 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.009237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.009237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.009379 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:21.009427 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.010109 (+ 682us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.010206 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:23:21.010658 7469 heartbeater.cc:347] Connected to a master server at 127.31.214.124:38083
I20260928 06:23:21.010757 7469 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:21.010985 7469 heartbeater.cc:511] Master 127.31.214.124:38083 requested a full tablet report, sending...
I20260928 06:23:21.011474 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.997090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:34981 (local address 127.31.214.67:56163)
0928 06:23:20.997733 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.997742 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.997797 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.998008 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.998009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.998320 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.998720 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.998723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.999700 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.999701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.001079 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.001082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.001625 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.001626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.001772 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.003008 (+ 1236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:21.003013 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:21.006624 (+ 3611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:21.008380 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:21.008381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:21.008381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:21.008671 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:21.008900 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:21.008901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:21.008901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:21.008962 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:21.009483 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:21.009484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:21.009526 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:23:21.010661 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0928 06:23:21.011098 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":61,"thread_start_us":59,"threads_started":1}
I20260928 06:23:21.011550 7017 ts_manager.cc:194] Registered new tserver with Master: ff725a15eb8f429eb9c4c76a8c46dafb (127.31.214.67:42441)
I20260928 06:23:21.011605 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.997249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:34249 (local address 127.31.214.126:46093)
0928 06:23:20.997298 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.997300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.997650 (+ 350us) server_negotiation.cc:407] Connection header received
0928 06:23:20.997959 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.997961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.997975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.998008 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:20.998777 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.999582 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.001192 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.001495 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.001534 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.007970 (+ 6436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:21.007972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:21.007972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:21.007980 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:21.010271 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.010777 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.010778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.010778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.010832 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.011017 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.011018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.011019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.011145 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:21.011187 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.011401 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.011491 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:23:21.011924 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.996883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:34249)
0928 06:23:20.997576 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:20.997587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:20.997858 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:20.998078 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:20.998079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:20.998324 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:20.998644 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:20.998663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:20.999700 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:20.999700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.001079 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.001082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.001623 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.001624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.001772 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.002742 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:21.002746 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:21.007856 (+ 5110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:21.010386 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:21.010387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:21.010388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:21.010688 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:21.010897 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:21.010898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:21.010898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:21.010948 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:21.011235 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:21.011236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:21.011279 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:23:21.011686 (+ 407us) client_negotiation.cc:240] Negotiation successful
0928 06:23:21.011789 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":169,"thread_start_us":62,"threads_started":1}
I20260928 06:23:21.012274 7470 heartbeater.cc:347] Connected to a master server at 127.31.214.126:46093
I20260928 06:23:21.012334 7470 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:21.012436 7471 heartbeater.cc:347] Connected to a master server at 127.31.214.125:34981
I20260928 06:23:21.012519 7471 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:21.012601 7471 heartbeater.cc:511] Master 127.31.214.125:34981 requested a full tablet report, sending...
I20260928 06:23:21.012710 7470 heartbeater.cc:511] Master 127.31.214.126:46093 requested a full tablet report, sending...
I20260928 06:23:21.012916 6829 ts_manager.cc:194] Registered new tserver with Master: ff725a15eb8f429eb9c4c76a8c46dafb (127.31.214.67:42441)
I20260928 06:23:21.013110 6231 ts_manager.cc:194] Re-registered known tserver with Master: ff725a15eb8f429eb9c4c76a8c46dafb (127.31.214.67:42441)
I20260928 06:23:21.013203 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.998593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.31.214.125:34981)
0928 06:23:20.999674 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.999676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:20.999684 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:23:20.999703 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:20.999704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:20.999716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:20.999786 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:21.000620 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.001336 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.002308 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.002630 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.002717 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.004154 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:21.004155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:21.004156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:21.004162 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:21.005851 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.006604 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.006605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.006606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.006663 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.007340 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.007341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.007342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.008380 (+ 1038us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:21.010251 (+ 1871us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.010307 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.010399 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":45,"threads_started":1}
I20260928 06:23:21.013622 6231 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:34249
I20260928 06:23:21.015144 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.999127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.31.214.124:38083)
0928 06:23:21.000589 (+ 1462us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.000591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.002529 (+ 1938us) server_negotiation.cc:407] Connection header received
0928 06:23:21.002614 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.002615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.002628 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.002666 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:21.003605 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.004487 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.005684 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.006047 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.006089 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.007629 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:21.007631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:21.007631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:21.007637 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:21.010083 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.013687 (+ 3604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.013688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.013688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.013993 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.014505 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.014506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.014507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.014655 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:21.014760 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.014916 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.015006 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":91,"threads_started":1}
I20260928 06:23:21.016566 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:20.998699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.31.214.126:46093)
0928 06:23:20.999616 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:20.999618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.002405 (+ 2787us) server_negotiation.cc:407] Connection header received
0928 06:23:21.002456 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.002457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.002471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.002548 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:21.004571 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.005468 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.011717 (+ 6249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.012026 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.012108 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.013915 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:21.013916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:21.013917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:21.013923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:21.015700 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.016021 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.016021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.016022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.016073 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.016208 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.016208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.016209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.016305 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:21.016373 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.016442 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.016504 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":116,"threads_started":1}
I20260928 06:23:21.018316 6231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:21.020005 6231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:21.027997 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.024198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.31.214.66:43369)
0928 06:23:21.024472 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.024479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.024493 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:23:21.024545 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.024549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.024568 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.024665 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:21.024835 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.025705 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.026942 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.027605 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.027692 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.027782 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.027851 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.027903 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260928 06:23:21.028234 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.024097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:43369 (local address 127.0.0.1:37226)
0928 06:23:21.024414 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:21.024426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:21.024483 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:21.024672 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:21.024673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:21.024676 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:21.024733 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.024735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.025811 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.025812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.026831 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.026833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.027699 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.027700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.027761 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.027766 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:23:21.028099 (+ 333us) client_negotiation.cc:240] Negotiation successful
0928 06:23:21.028163 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":163,"thread_start_us":99,"threads_started":1}
I20260928 06:23:21.028577 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.024098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:42441 (local address 127.0.0.1:47482)
0928 06:23:21.024138 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:21.024151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:21.024195 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:21.024744 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:21.024746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:21.024747 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:21.024793 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.024795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.026330 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.026331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.027361 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.027364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.028310 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.028311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.028414 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.028417 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:23:21.028450 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 06:23:21.028495 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":30}
I20260928 06:23:21.028954 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.024392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.31.214.67:42441)
0928 06:23:21.024552 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.024556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.024567 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:23:21.024605 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.024608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.024623 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.024879 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:21.024937 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.026228 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.027449 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.028196 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.028707 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.028798 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.028827 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.028873 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260928 06:23:21.029086 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 4448aa819c1a42f88c232951a163948f (DEFAULT_TABLE table=test-table [id=e6868e6cd8c940589e8962ca750bdf64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:21.029484 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448aa819c1a42f88c232951a163948f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:21.029786 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.024149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:36869 (local address 127.0.0.1:58952)
0928 06:23:21.024303 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:21.024314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:21.024366 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:21.024994 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:21.024995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:21.024997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:21.025036 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.025037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.026463 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.026464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.028435 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.028438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.029492 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.029494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.029566 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.029570 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:23:21.029635 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:23:21.029681 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260928 06:23:21.030200 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.024573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.31.214.65:36869)
0928 06:23:21.024809 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.024813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.024824 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:23:21.024866 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.024868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.024885 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.025111 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:21.025170 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.026377 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.028570 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.029411 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.029483 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.029555 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.029998 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.030113 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260928 06:23:21.030987 7130 tablet_service.cc:1467] Processing CreateTablet for tablet 4448aa819c1a42f88c232951a163948f (DEFAULT_TABLE table=test-table [id=e6868e6cd8c940589e8962ca750bdf64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:21.031409 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448aa819c1a42f88c232951a163948f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:21.033713 7495 tablet_bootstrap.cc:492] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: Bootstrap starting.
I20260928 06:23:21.034442 7495 tablet_bootstrap.cc:654] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:21.034503 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 4448aa819c1a42f88c232951a163948f (DEFAULT_TABLE table=test-table [id=e6868e6cd8c940589e8962ca750bdf64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:21.034801 7495 log.cc:824] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:21.034897 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448aa819c1a42f88c232951a163948f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:21.035517 7497 tablet_bootstrap.cc:492] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: Bootstrap starting.
I20260928 06:23:21.035686 7495 tablet_bootstrap.cc:492] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: No bootstrap required, opened a new log
I20260928 06:23:21.035796 7495 ts_tablet_manager.cc:1397] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:21.036164 7497 tablet_bootstrap.cc:654] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:21.036494 7497 log.cc:824] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:21.037436 7497 tablet_bootstrap.cc:492] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: No bootstrap required, opened a new log
I20260928 06:23:21.037515 7497 ts_tablet_manager.cc:1397] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:21.038379 7499 tablet_bootstrap.cc:492] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: Bootstrap starting.
I20260928 06:23:21.038561 7495 raft_consensus.cc:348] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } }
I20260928 06:23:21.038761 7495 raft_consensus.cc:374] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:21.038813 7495 raft_consensus.cc:729] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e4de63f2044696abbf32f9687fb04c, State: Initialized, Role: FOLLOWER
I20260928 06:23:21.038959 7495 consensus_queue.cc:260] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } }
I20260928 06:23:21.039016 7499 tablet_bootstrap.cc:654] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:21.039242 7495 ts_tablet_manager.cc:1428] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:23:21.039389 7334 heartbeater.cc:503] Master 127.31.214.126:46093 was elected leader, sending a full tablet report...
I20260928 06:23:21.039793 7499 log.cc:824] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:21.041028 7499 tablet_bootstrap.cc:492] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: No bootstrap required, opened a new log
I20260928 06:23:21.041082 7499 ts_tablet_manager.cc:1397] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:23:21.041004 7497 raft_consensus.cc:348] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } }
I20260928 06:23:21.041180 7497 raft_consensus.cc:374] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:21.041234 7497 raft_consensus.cc:729] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edcac2b2bcd1470ba1bd11588faec84b, State: Initialized, Role: FOLLOWER
I20260928 06:23:21.041360 7497 consensus_queue.cc:260] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } }
I20260928 06:23:21.041687 7197 heartbeater.cc:503] Master 127.31.214.126:46093 was elected leader, sending a full tablet report...
I20260928 06:23:21.041678 7497 ts_tablet_manager.cc:1428] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:23:21.044067 7499 raft_consensus.cc:348] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } }
I20260928 06:23:21.044188 7499 raft_consensus.cc:374] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:21.044210 7499 raft_consensus.cc:729] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff725a15eb8f429eb9c4c76a8c46dafb, State: Initialized, Role: FOLLOWER
I20260928 06:23:21.044328 7499 consensus_queue.cc:260] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } }
I20260928 06:23:21.044624 7499 ts_tablet_manager.cc:1428] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:23:21.044677 7470 heartbeater.cc:503] Master 127.31.214.126:46093 was elected leader, sending a full tablet report...
I20260928 06:23:21.086911 7503 raft_consensus.cc:482] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:21.087049 7503 raft_consensus.cc:504] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } }
I20260928 06:23:21.087421 7503 leader_election.cc:288] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8e4de63f2044696abbf32f9687fb04c (127.31.214.66:43369), edcac2b2bcd1470ba1bd11588faec84b (127.31.214.65:36869)
I20260928 06:23:21.091790 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.087817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:35943 (local address 127.31.214.66:43369)
0928 06:23:21.087915 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.087918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.087927 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:23:21.087953 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.087955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.087965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.088023 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:21.088271 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.089086 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.090719 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.091428 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.091500 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.091509 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.091652 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.091699 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:23:21.091789 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.087744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:60267 (local address 127.31.214.65:36869)
0928 06:23:21.087796 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.087799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.087810 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:23:21.087842 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.087843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.087854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.087914 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:21.088243 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.089102 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.090712 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.091366 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.091459 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.091471 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.091664 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.091712 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:23:21.091800 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.087583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:36869 (local address 127.31.214.67:60267)
0928 06:23:21.087673 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:21.087682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:21.087732 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:21.087919 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:21.087920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:21.087921 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:21.088024 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.088027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.089197 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.089197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.090596 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.090600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.091458 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.091459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.091562 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.091567 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:23:21.091627 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:23:21.091677 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":83}
I20260928 06:23:21.091814 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.087575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:43369 (local address 127.31.214.67:35943)
0928 06:23:21.087640 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:21.087654 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:21.087704 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:21.088044 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:21.088045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:21.088047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:21.088095 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.088096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.089194 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.089195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.090596 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:21.090600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.091501 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.091501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:21.091568 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.091570 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:23:21.091627 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:23:21.091677 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":87}
I20260928 06:23:21.092160 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448aa819c1a42f88c232951a163948f" candidate_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e4de63f2044696abbf32f9687fb04c" is_pre_election: true
I20260928 06:23:21.092231 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448aa819c1a42f88c232951a163948f" candidate_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edcac2b2bcd1470ba1bd11588faec84b" is_pre_election: true
I20260928 06:23:21.092332 7287 raft_consensus.cc:2454] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff725a15eb8f429eb9c4c76a8c46dafb in term 0.
I20260928 06:23:21.092444 7150 raft_consensus.cc:2454] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff725a15eb8f429eb9c4c76a8c46dafb in term 0.
I20260928 06:23:21.092597 7357 leader_election.cc:302] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e4de63f2044696abbf32f9687fb04c, ff725a15eb8f429eb9c4c76a8c46dafb; no voters:
I20260928 06:23:21.092808 7503 raft_consensus.cc:2781] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:21.092916 7503 raft_consensus.cc:482] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:21.092955 7503 raft_consensus.cc:3037] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:21.093842 7503 raft_consensus.cc:504] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } }
I20260928 06:23:21.094065 7503 leader_election.cc:288] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [CANDIDATE]: Term 1 election: Requested vote from peers d8e4de63f2044696abbf32f9687fb04c (127.31.214.66:43369), edcac2b2bcd1470ba1bd11588faec84b (127.31.214.65:36869)
I20260928 06:23:21.094211 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448aa819c1a42f88c232951a163948f" candidate_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e4de63f2044696abbf32f9687fb04c"
I20260928 06:23:21.094287 7150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448aa819c1a42f88c232951a163948f" candidate_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edcac2b2bcd1470ba1bd11588faec84b"
I20260928 06:23:21.094305 7287 raft_consensus.cc:3037] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:21.094374 7150 raft_consensus.cc:3037] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:21.095166 7287 raft_consensus.cc:2454] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff725a15eb8f429eb9c4c76a8c46dafb in term 1.
I20260928 06:23:21.095278 7150 raft_consensus.cc:2454] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff725a15eb8f429eb9c4c76a8c46dafb in term 1.
W20260928 06:23:21.095358 7336 tablet.cc:2369] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:23:21.095446 7358 leader_election.cc:302] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edcac2b2bcd1470ba1bd11588faec84b, ff725a15eb8f429eb9c4c76a8c46dafb; no voters:
I20260928 06:23:21.095541 7503 raft_consensus.cc:2781] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:21.095695 7503 raft_consensus.cc:686] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 1 LEADER]: Becoming Leader. State: Replica: ff725a15eb8f429eb9c4c76a8c46dafb, State: Running, Role: LEADER
I20260928 06:23:21.095810 7503 consensus_queue.cc:237] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } }
I20260928 06:23:21.096851 6231 catalog_manager.cc:5495] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb reported cstate change: term changed from 0 to 1, leader changed from <none> to ff725a15eb8f429eb9c4c76a8c46dafb (127.31.214.67). New cstate: current_term: 1 leader_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff725a15eb8f429eb9c4c76a8c46dafb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 42441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:21.108304 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.105620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.31.214.67:42441)
0928 06:23:21.105689 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.105691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.105701 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:23:21.105744 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.105745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.105763 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.105852 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:21.106033 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.106560 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.107416 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.107665 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.107746 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.107947 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:21.108058 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:21.108159 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.108185 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.108209 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:23:21.110591 7150 raft_consensus.cc:1261] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [term 1 FOLLOWER]: Refusing update from remote peer ff725a15eb8f429eb9c4c76a8c46dafb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:21.110616 7287 raft_consensus.cc:1261] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [term 1 FOLLOWER]: Refusing update from remote peer ff725a15eb8f429eb9c4c76a8c46dafb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:21.110987 7503 consensus_queue.cc:1035] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [LEADER]: Connected to new peer: Peer: permanent_uuid: "edcac2b2bcd1470ba1bd11588faec84b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:21.111162 7504 consensus_queue.cc:1035] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8e4de63f2044696abbf32f9687fb04c" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:21.113871 7507 mvcc.cc:205] Tried to move back new op lower bound from 7334201758145585152 to 7334201758089166848. Current Snapshot: MvccSnapshot[applied={T|T < 7334201758145585152}]
I20260928 06:23:21.114789 7509 mvcc.cc:205] Tried to move back new op lower bound from 7334201758145585152 to 7334201758089166848. Current Snapshot: MvccSnapshot[applied={T|T < 7334201758145585152}]
I20260928 06:23:21.115938 7508 mvcc.cc:205] Tried to move back new op lower bound from 7334201758145585152 to 7334201758089166848. Current Snapshot: MvccSnapshot[applied={T|T < 7334201758145585152}]
I20260928 06:23:21.116992 6231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:21.117087 6231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:21.118741 6231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d549ccb5ce04c039d00ed910205c763: Sending DeleteTablet for 3 replicas of tablet 4448aa819c1a42f88c232951a163948f
I20260928 06:23:21.119092 7130 tablet_service.cc:1514] Processing DeleteTablet for tablet 4448aa819c1a42f88c232951a163948f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:21 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:58952
I20260928 06:23:21.119287 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 4448aa819c1a42f88c232951a163948f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:21 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:47482
I20260928 06:23:21.119310 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 4448aa819c1a42f88c232951a163948f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:21 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:37226
I20260928 06:23:21.119503 7518 tablet_replica.cc:330] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: stopping tablet replica
I20260928 06:23:21.119519 7519 tablet_replica.cc:330] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: stopping tablet replica
I20260928 06:23:21.119612 7519 raft_consensus.cc:2229] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:21.119609 7518 raft_consensus.cc:2229] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:23:21.119710 7518 raft_consensus.cc:2258] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:21.119714 7519 raft_consensus.cc:2258] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:21.120206 7518 ts_tablet_manager.cc:1905] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:21.120496 7519 ts_tablet_manager.cc:1905] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:21.120965 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 6199
I20260928 06:23:21.122874 7519 ts_tablet_manager.cc:1918] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:21.122951 7519 log.cc:1196] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/4448aa819c1a42f88c232951a163948f
I20260928 06:23:21.122934 7518 ts_tablet_manager.cc:1918] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:21.123008 7518 log.cc:1196] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/4448aa819c1a42f88c232951a163948f
I20260928 06:23:21.123250 7519 ts_tablet_manager.cc:1939] T 4448aa819c1a42f88c232951a163948f P d8e4de63f2044696abbf32f9687fb04c: Deleting consensus metadata
I20260928 06:23:21.123329 7518 ts_tablet_manager.cc:1939] T 4448aa819c1a42f88c232951a163948f P ff725a15eb8f429eb9c4c76a8c46dafb: Deleting consensus metadata
I20260928 06:23:21.126458 7517 tablet_replica.cc:330] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: stopping tablet replica
I20260928 06:23:21.126672 7517 raft_consensus.cc:2229] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:21.126901 7517 raft_consensus.cc:2258] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 06:23:21.128315 7222 connection.cc:535] server connection from 127.0.0.1:37226 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:23:21.128430 7358 connection.cc:535] server connection from 127.0.0.1:47482 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:23:21.129423 7517 ts_tablet_manager.cc:1905] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:21.131479 7517 ts_tablet_manager.cc:1918] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:21.131569 7517 log.cc:1196] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/4448aa819c1a42f88c232951a163948f
I20260928 06:23:21.131892 7517 ts_tablet_manager.cc:1939] T 4448aa819c1a42f88c232951a163948f P edcac2b2bcd1470ba1bd11588faec84b: Deleting consensus metadata
W20260928 06:23:21.132285 7084 connection.cc:441] server connection from 127.0.0.1:58952 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58952 (request call id 1) could send its response
I20260928 06:23:21.136200 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.120055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.31.214.124:38083)
0928 06:23:21.120100 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.120102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.126310 (+ 6208us) server_negotiation.cc:407] Connection header received
0928 06:23:21.126394 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.126395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.126414 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.126493 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:21.127477 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.128296 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.129568 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.129851 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.129904 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.132423 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:21.132424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:21.132425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:21.132433 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:21.134809 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.135230 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.135231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.135232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.135292 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.135509 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.135510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.135510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.135654 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:21.135738 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.135959 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.136060 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:23:21.140071 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.130399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.31.214.124:38083)
0928 06:23:21.130446 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.130448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.131082 (+ 634us) server_negotiation.cc:407] Connection header received
0928 06:23:21.131199 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.131200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.131212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.131245 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:21.132927 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.133434 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.134950 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.135292 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.135333 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.137032 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:21.137034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:21.137034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:21.137041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:21.138785 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.139205 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.139206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.139207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.139262 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.139481 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.139482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.139483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.139629 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:21.139726 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.139831 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.139906 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:23:21.140545 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:21.130722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.31.214.125:34981)
0928 06:23:21.130785 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:21.130788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:21.132084 (+ 1296us) server_negotiation.cc:407] Connection header received
0928 06:23:21.132198 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:21.132199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:21.132215 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:21.132248 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:21.132963 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.133824 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.135084 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:21.135450 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:21.135495 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:21.136988 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:21.136990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:21.136990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:21.136998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:21.138841 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.139508 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.139509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.139509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.139563 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:21.139807 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:21.139808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:21.139809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:21.139954 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:21.140029 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:21.140346 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 06:23:21.140444 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:23:22.115995 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:22.115696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:34069)
0928 06:23:22.115837 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:22.115937 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260928 06:23:22.116187 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:22.115802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:42751)
0928 06:23:22.115969 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:22.116059 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260928 06:23:22.116376 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:22.116087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:46443)
0928 06:23:22.116257 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:22.116332 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
W20260928 06:23:22.116390 7334 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:46093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
W20260928 06:23:22.116434 7197 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:46093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
W20260928 06:23:22.116585 7470 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:46093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
I20260928 06:23:22.619504 7531 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d549ccb5ce04c039d00ed910205c763)
I20260928 06:23:22.619638 7531 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:22.619863 7531 leader_election.cc:288] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093)
I20260928 06:23:22.620256 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:22.620047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43566)
0928 06:23:22.620155 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:22.620218 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":65,"threads_started":1}
W20260928 06:23:22.620460 6815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
I20260928 06:23:22.620891 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:22.620786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43580)
0928 06:23:22.620840 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:22.620869 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:23:22.621025 6815 leader_election.cc:334] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093): Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
I20260928 06:23:22.621116 6815 leader_election.cc:302] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b0014cf7b5634653bacf2358884671a6; no voters: 8d549ccb5ce04c039d00ed910205c763
I20260928 06:23:22.621246 7531 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:23:23.117079 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:23.116815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:44485)
0928 06:23:23.116989 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:23.117030 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260928 06:23:23.117070 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:23.116821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:59625)
0928 06:23:23.116954 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:23.117006 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 06:23:23.117316 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:23.117099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:49701)
0928 06:23:23.117231 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:23.117272 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":47,"threads_started":1}
I20260928 06:23:24.118008 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:24.117724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:40887)
0928 06:23:24.117875 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:24.117918 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260928 06:23:24.118242 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:24.117913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:43419)
0928 06:23:24.118065 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:24.118190 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260928 06:23:24.118381 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:24.118027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:44285)
0928 06:23:24.118267 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:24.118333 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":57,"threads_started":1}
I20260928 06:23:24.848565 7540 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d549ccb5ce04c039d00ed910205c763)
I20260928 06:23:24.848664 7540 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:24.848821 7540 leader_election.cc:288] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093)
I20260928 06:23:24.849218 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:24.849021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43886)
0928 06:23:24.849142 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:24.849180 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260928 06:23:24.849689 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:24.849541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:43890)
0928 06:23:24.849610 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:24.849659 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:23:24.849776 6815 leader_election.cc:334] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093): Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
I20260928 06:23:24.849835 6815 leader_election.cc:302] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b0014cf7b5634653bacf2358884671a6; no voters: 8d549ccb5ce04c039d00ed910205c763
I20260928 06:23:24.849931 7540 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:23:25.118800 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:25.118548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:58009)
0928 06:23:25.118691 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:25.118736 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260928 06:23:25.119153 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:25.118892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:47283)
0928 06:23:25.119065 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:25.119108 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":53,"threads_started":1}
I20260928 06:23:25.119175 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:25.118892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:52325)
0928 06:23:25.119074 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:25.119115 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":56,"threads_started":1}
I20260928 06:23:26.119853 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:26.119593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:37777)
0928 06:23:26.119740 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:26.119790 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260928 06:23:26.120245 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:26.119978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:32799)
0928 06:23:26.120151 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:26.120190 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":51,"threads_started":1}
I20260928 06:23:26.120548 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:26.119576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:39121)
0928 06:23:26.120027 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:26.120066 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":68,"threads_started":1}
I20260928 06:23:27.120915 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:27.120601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:42759)
0928 06:23:27.120802 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:27.120853 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260928 06:23:27.121244 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:27.120894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:51115)
0928 06:23:27.121166 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:27.121203 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":54,"threads_started":1}
I20260928 06:23:27.121088 7551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:27.120895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:46051)
0928 06:23:27.121005 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:27.121038 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260928 06:23:27.736320 7552 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d549ccb5ce04c039d00ed910205c763)
I20260928 06:23:27.736425 7552 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:27.736588 7552 leader_election.cc:288] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093)
I20260928 06:23:27.736990 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:27.736765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:44156)
0928 06:23:27.736898 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:27.736948 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
W20260928 06:23:27.737185 6815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260928 06:23:27.737638 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:27.737513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:44168)
0928 06:23:27.737567 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:27.737612 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:23:27.737746 6815 leader_election.cc:334] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093): Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
I20260928 06:23:27.737831 6815 leader_election.cc:302] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b0014cf7b5634653bacf2358884671a6; no voters: 8d549ccb5ce04c039d00ed910205c763
I20260928 06:23:27.737974 7552 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:23:28.122148 7556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:28.121866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:50897)
0928 06:23:28.122023 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:28.122086 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":60,"threads_started":1}
I20260928 06:23:28.122200 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:28.121751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:56713)
0928 06:23:28.122121 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:28.122156 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":71,"threads_started":1}
I20260928 06:23:28.122387 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:28.121886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:38231)
0928 06:23:28.122283 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:28.122329 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":207,"threads_started":1}
I20260928 06:23:29.123286 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:29.122749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:36471)
0928 06:23:29.123176 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:29.123223 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":68,"threads_started":1}
I20260928 06:23:29.123373 7560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:29.122972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:44755)
0928 06:23:29.123302 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:29.123333 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":159,"threads_started":1}
I20260928 06:23:29.123608 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:29.122987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:48549)
0928 06:23:29.123513 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:29.123563 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":61,"threads_started":1}
I20260928 06:23:30.124212 7562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:30.123973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:34033)
0928 06:23:30.124102 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:30.124150 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260928 06:23:30.124384 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:30.124011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:44337)
0928 06:23:30.124299 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:30.124339 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20260928 06:23:30.124466 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:30.123834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:54995)
0928 06:23:30.124370 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:30.124410 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":86,"threads_started":1}
I20260928 06:23:31.125430 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:31.125250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:47973)
0928 06:23:31.125349 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:31.125390 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
I20260928 06:23:31.125411 7565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:31.125100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:38997)
0928 06:23:31.125299 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:31.125349 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
I20260928 06:23:31.125411 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:31.124970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:32845)
0928 06:23:31.125332 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:31.125369 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":68,"threads_started":1}
I20260928 06:23:32.126174 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:32.125899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:54105)
0928 06:23:32.126069 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:32.126113 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":121,"threads_started":1}
I20260928 06:23:32.126267 7568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:32.125984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:54517)
0928 06:23:32.126151 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:32.126194 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":59,"threads_started":1}
I20260928 06:23:32.126407 7569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:32.126142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:48603)
0928 06:23:32.126276 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:32.126312 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260928 06:23:32.670284 7570 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d549ccb5ce04c039d00ed910205c763)
I20260928 06:23:32.670383 7570 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:32.670560 7570 leader_election.cc:288] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093)
I20260928 06:23:32.671021 7571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:32.670754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:35276)
0928 06:23:32.670887 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:32.670939 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260928 06:23:32.671571 7571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:32.671433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:35290)
0928 06:23:32.671505 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:32.671538 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 06:23:32.671656 6815 leader_election.cc:334] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093): Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
I20260928 06:23:32.671703 6815 leader_election.cc:302] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b0014cf7b5634653bacf2358884671a6; no voters: 8d549ccb5ce04c039d00ed910205c763
I20260928 06:23:32.671777 7570 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:23:33.127206 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:33.126868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:48469)
0928 06:23:33.127096 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:33.127141 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":61,"threads_started":1}
I20260928 06:23:33.127250 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:33.126854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:60791)
0928 06:23:33.127146 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:33.127184 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260928 06:23:33.127290 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:33.126937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:33031)
0928 06:23:33.127201 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:33.127240 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":65,"threads_started":1}
I20260928 06:23:34.128059 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:34.127754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:41513)
0928 06:23:34.127975 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:34.128015 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":76,"threads_started":1}
I20260928 06:23:34.128070 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:34.127754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:50975)
0928 06:23:34.127966 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:34.128009 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":64,"threads_started":1}
I20260928 06:23:34.128207 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:34.128012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:41485)
0928 06:23:34.128130 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:34.128167 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":51,"threads_started":1}
I20260928 06:23:35.128823 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:35.128545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:56315)
0928 06:23:35.128719 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:35.128763 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260928 06:23:35.128897 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:35.128517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:47733)
0928 06:23:35.128789 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:35.128840 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260928 06:23:35.128986 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:35.128802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:60919)
0928 06:23:35.128911 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:35.128942 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":47,"threads_started":1}
I20260928 06:23:36.129815 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:36.129543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:52669)
0928 06:23:36.129703 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:36.129752 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260928 06:23:36.129827 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:36.129612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:56331)
0928 06:23:36.129738 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:36.129775 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260928 06:23:36.130089 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:36.129728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:42523)
0928 06:23:36.129972 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:36.130014 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":49,"threads_started":1}
I20260928 06:23:37.130537 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:37.130257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:38545)
0928 06:23:37.130425 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:37.130471 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260928 06:23:37.130692 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:37.130471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:34195)
0928 06:23:37.130600 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:37.130639 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":53,"threads_started":1}
I20260928 06:23:37.130831 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:37.130391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:45347)
0928 06:23:37.130746 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:37.130788 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":48,"threads_started":1}
I20260928 06:23:38.131433 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:38.131179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:44381)
0928 06:23:38.131328 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:38.131368 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260928 06:23:38.131570 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:38.131361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:44457)
0928 06:23:38.131468 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:38.131514 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":48,"threads_started":1}
I20260928 06:23:38.131649 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:38.131333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:37421)
0928 06:23:38.131512 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:38.131606 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":58,"threads_started":1}
I20260928 06:23:39.016993 7591 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d549ccb5ce04c039d00ed910205c763)
I20260928 06:23:39.017223 7591 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:39.017463 7591 leader_election.cc:288] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093)
I20260928 06:23:39.017894 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:39.017699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:45000)
0928 06:23:39.017817 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:39.017855 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
W20260928 06:23:39.018038 6815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260928 06:23:39.018416 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:39.018309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:45002)
0928 06:23:39.018359 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:39.018386 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:23:39.018491 6815 leader_election.cc:334] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093): Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
I20260928 06:23:39.018527 6815 leader_election.cc:302] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b0014cf7b5634653bacf2358884671a6; no voters: 8d549ccb5ce04c039d00ed910205c763
I20260928 06:23:39.018608 7591 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:23:39.132299 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:39.132138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:41221)
0928 06:23:39.132240 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:39.132265 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
I20260928 06:23:39.132299 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:39.131894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:56345)
0928 06:23:39.132190 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:39.132240 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":64,"threads_started":1}
I20260928 06:23:39.132320 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:39.131991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:59877)
0928 06:23:39.132226 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:39.132269 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":75,"threads_started":1}
I20260928 06:23:40.133111 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:40.132867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:34147)
0928 06:23:40.133014 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:40.133055 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
I20260928 06:23:40.133157 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:40.132892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:49195)
0928 06:23:40.133080 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:40.133113 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":43,"threads_started":1}
I20260928 06:23:40.133098 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:40.132864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:57291)
0928 06:23:40.132995 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:40.133039 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260928 06:23:41.133919 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:41.133649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:38607)
0928 06:23:41.133795 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:41.133850 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260928 06:23:41.134291 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:41.134020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:60235)
0928 06:23:41.134170 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:41.134209 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
I20260928 06:23:41.134333 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:41.133938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:50521)
0928 06:23:41.134247 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:41.134286 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":63,"threads_started":1}
I20260928 06:23:41.977634 7603 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d549ccb5ce04c039d00ed910205c763)
I20260928 06:23:41.977742 7603 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:41.977929 7603 leader_election.cc:288] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093)
I20260928 06:23:41.978340 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:41.978136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:45132)
0928 06:23:41.978248 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:41.978299 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260928 06:23:41.978840 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:41.978726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:45134)
0928 06:23:41.978786 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:41.978817 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:23:41.979040 6815 leader_election.cc:334] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093): Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
I20260928 06:23:41.979130 6815 leader_election.cc:302] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b0014cf7b5634653bacf2358884671a6; no voters: 8d549ccb5ce04c039d00ed910205c763
I20260928 06:23:41.979328 7603 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:23:42.135018 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:42.134624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:45173)
0928 06:23:42.134914 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:42.134970 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":65,"threads_started":1}
I20260928 06:23:42.135243 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:42.134909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:50367)
0928 06:23:42.135158 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:42.135195 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":58,"threads_started":1}
I20260928 06:23:42.135324 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:42.134984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:44049)
0928 06:23:42.135250 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:42.135282 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":43,"threads_started":1}
I20260928 06:23:43.136281 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:43.135791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:36689)
0928 06:23:43.136169 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:43.136218 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":72,"threads_started":1}
I20260928 06:23:43.136412 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:43.136096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:38259)
0928 06:23:43.136321 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:43.136363 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":59,"threads_started":1}
I20260928 06:23:43.136448 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:43.135797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:36891)
0928 06:23:43.136348 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:43.136408 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":71,"threads_started":1}
I20260928 06:23:44.137104 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:44.136823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:54769)
0928 06:23:44.136976 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:44.137023 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260928 06:23:44.137369 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:44.137176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:47135)
0928 06:23:44.137282 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:44.137322 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":47,"threads_started":1}
I20260928 06:23:44.137388 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:44.137015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:33721)
0928 06:23:44.137298 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:44.137339 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":56,"threads_started":1}
I20260928 06:23:45.138180 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:45.137832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:37083)
0928 06:23:45.138079 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:45.138124 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":63,"threads_started":1}
I20260928 06:23:45.138175 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:45.137832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:47947)
0928 06:23:45.138071 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:45.138118 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":66,"threads_started":1}
I20260928 06:23:45.138406 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:45.137773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:51209)
0928 06:23:45.138071 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:45.138119 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":70,"threads_started":1}
I20260928 06:23:46.139107 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:46.138855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:59811)
0928 06:23:46.138998 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:46.139042 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260928 06:23:46.139178 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:46.138946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:56269)
0928 06:23:46.139079 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:46.139121 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":53,"threads_started":1}
I20260928 06:23:46.139385 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:46.138895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:38531)
0928 06:23:46.139297 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:46.139333 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":63,"threads_started":1}
I20260928 06:23:46.512272 7621 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d549ccb5ce04c039d00ed910205c763)
I20260928 06:23:46.512382 7621 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8d549ccb5ce04c039d00ed910205c763" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 46093 } } peers { permanent_uuid: "b0014cf7b5634653bacf2358884671a6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 34981 } attrs { promote: false } } peers { permanent_uuid: "9bc67401c85c4bb88ae6abd1824f6cc2" member_type: NON_VOTER last_known_addr { host: "127.31.214.124" port: 38083 } attrs { promote: true } }
I20260928 06:23:46.512563 7621 leader_election.cc:288] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093)
I20260928 06:23:46.512974 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:46.512717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:45360)
0928 06:23:46.512825 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:46.512872 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
W20260928 06:23:46.513125 6815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260928 06:23:46.513609 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:46.513497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.0.0.1:45370)
0928 06:23:46.513551 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:46.513579 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:23:46.513724 6815 leader_election.cc:334] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d549ccb5ce04c039d00ed910205c763 (127.31.214.126:46093): Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
I20260928 06:23:46.513808 6815 leader_election.cc:302] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b0014cf7b5634653bacf2358884671a6; no voters: 8d549ccb5ce04c039d00ed910205c763
I20260928 06:23:46.514000 7621 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b0014cf7b5634653bacf2358884671a6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:23:47.139907 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:47.139626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:50471)
0928 06:23:47.139802 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:47.139846 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260928 06:23:47.140014 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:47.139821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:38133)
0928 06:23:47.139936 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:47.139964 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260928 06:23:47.140050 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:47.139836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:47473)
0928 06:23:47.139957 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:47.139994 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
I20260928 06:23:48.140805 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:48.140458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:47403)
0928 06:23:48.140707 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:48.140749 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":67,"threads_started":1}
I20260928 06:23:48.140805 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:48.140532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:44027)
0928 06:23:48.140707 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:48.140750 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260928 06:23:48.140820 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:48.140532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:54307)
0928 06:23:48.140686 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:48.140752 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260928 06:23:49.141688 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:49.141382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:57311)
0928 06:23:49.141580 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:49.141619 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":61,"threads_started":1}
I20260928 06:23:49.141916 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:49.141526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:44495)
0928 06:23:49.141826 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:49.141865 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":62,"threads_started":1}
I20260928 06:23:49.142206 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:49.141402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:37255)
0928 06:23:49.142125 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:49.142163 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":53,"threads_started":1}
I20260928 06:23:50.142616 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:50.142371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:48719)
0928 06:23:50.142505 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:50.142554 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260928 06:23:50.142762 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:50.142513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:50223)
0928 06:23:50.142671 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:50.142712 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":56,"threads_started":1}
I20260928 06:23:50.142885 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:50.142647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:47783)
0928 06:23:50.142795 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:50.142840 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 06:23:51.143498 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:51.143181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.66:56219)
0928 06:23:51.143396 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:51.143438 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":71,"threads_started":1}
I20260928 06:23:51.143819 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:51.143459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.65:58449)
0928 06:23:51.143755 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:51.143783 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":56,"threads_started":1}
I20260928 06:23:51.143843 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:51.143567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:46093 (local address 127.31.214.67:40011)
0928 06:23:51.143715 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:51.143753 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:46093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":54,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.31.214.126:46093,127.31.214.125:34981,127.31.214.124:38083, num_attempts: 1)
I20260928 06:23:51.367799 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 6977
I20260928 06:23:51.374322 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 7204
I20260928 06:23:51.380311 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 7340
I20260928 06:23:51.386539 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 6695
I20260928 06:23:51.392133 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 6883
2026-09-28T06:23:51Z chronyd exiting
I20260928 06:23:51.408651 32601 test_util.cc:175] -----------------------------------------------
I20260928 06:23:51.408715 32601 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790576570121568-32601-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33276 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:51 dist-test-slave-q03f krb5kdc[7641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:51 dist-test-slave-q03f krb5kdc[7641](info): set up 2 sockets
Sep 28 06:23:51 dist-test-slave-q03f krb5kdc[7641](info): commencing operation
krb5kdc: starting...
W20260928 06:23:52.474599 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:52 dist-test-slave-q03f krb5kdc[7641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.126.keytab.
Entry for principal oryx/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.126.keytab.
I20260928 06:23:52.605629 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:37929
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.126.keytab
--principal=oryx/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:37929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:52.689852 7654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:52.690133 7654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:52.690171 7654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:52.691316 7654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:52.691390 7654 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:23:52.691411 7654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:52.691443 7654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:52.691471 7654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:52.691497 7654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:52.693032 7654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:37929
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.126.keytab
--principal=oryx/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:37929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:52.693316 7654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:52.693554 7654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:52.699389 7660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:52.699451 7654 server_base.cc:1034] running on GCE node
W20260928 06:23:52.699399 7662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:52.699494 7659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:52.699828 7654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:52.700133 7654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:52.701361 7654 hybrid_clock.cc:648] HybridClock initialized: now 1790576632701301 us; error 67 us; skew 500 ppm
Sep 28 06:23:52 dist-test-slave-q03f krb5kdc[7641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576632, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:52.704373 7654 init.cc:377] Logged in from keytab as oryx/127.31.214.126@KRBTEST.COM (short username oryx)
I20260928 06:23:52.704855 7654 webserver.cc:460] Webserver started at http://127.31.214.126:33667/ using document root <none> and password file <none>
I20260928 06:23:52.705096 7654 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:52.705142 7654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:52.705238 7654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:52.706393 7654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "97a2f9fafc2643cfbc3094e29f2b24e2"
format_stamp: "Formatted at 2026-09-28 06:23:52 on dist-test-slave-q03f"
server_key: "c0b3edfdf1a6d290c389959311a277da"
server_key_iv: "8d2fb3fcf9d38041690096fe63d2f1aa"
server_key_version: "encryptionkey@0"
I20260928 06:23:52.706705 7654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "97a2f9fafc2643cfbc3094e29f2b24e2"
format_stamp: "Formatted at 2026-09-28 06:23:52 on dist-test-slave-q03f"
server_key: "c0b3edfdf1a6d290c389959311a277da"
server_key_iv: "8d2fb3fcf9d38041690096fe63d2f1aa"
server_key_version: "encryptionkey@0"
I20260928 06:23:52.708123 7654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:23:52.709023 7669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:52.709240 7654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:52.709321 7654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "97a2f9fafc2643cfbc3094e29f2b24e2"
format_stamp: "Formatted at 2026-09-28 06:23:52 on dist-test-slave-q03f"
server_key: "c0b3edfdf1a6d290c389959311a277da"
server_key_iv: "8d2fb3fcf9d38041690096fe63d2f1aa"
server_key_version: "encryptionkey@0"
I20260928 06:23:52.709405 7654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:52.719261 7654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:52.721587 7654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:52.721753 7654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:52.726658 7654 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:37929
I20260928 06:23:52.726768 7731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:37929 every 8 connection(s)
I20260928 06:23:52.727101 7654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:52.728004 7732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:52.730690 7732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2: Bootstrap starting.
I20260928 06:23:52.731070 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 7654
I20260928 06:23:52.731206 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:52.731463 32601 external_mini_cluster.cc:1428] Setting key ea99c7d7db8cf8bae9a3bfb93b885df0
I20260928 06:23:52.731640 7732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:52.732029 7732 log.cc:824] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:52.733130 7732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2: No bootstrap required, opened a new log
I20260928 06:23:52.735808 7732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a2f9fafc2643cfbc3094e29f2b24e2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37929 } }
I20260928 06:23:52.736012 7732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:52.736048 7732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a2f9fafc2643cfbc3094e29f2b24e2, State: Initialized, Role: FOLLOWER
I20260928 06:23:52.736212 7732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a2f9fafc2643cfbc3094e29f2b24e2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37929 } }
I20260928 06:23:52.736315 7732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:52.736356 7732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:52.736415 7732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:52.737154 7732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a2f9fafc2643cfbc3094e29f2b24e2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37929 } }
I20260928 06:23:52.737296 7732 leader_election.cc:302] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97a2f9fafc2643cfbc3094e29f2b24e2; no voters:
I20260928 06:23:52.737493 7732 leader_election.cc:288] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:52.737547 7737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:52.737722 7737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [term 1 LEADER]: Becoming Leader. State: Replica: 97a2f9fafc2643cfbc3094e29f2b24e2, State: Running, Role: LEADER
I20260928 06:23:52.737812 7732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:52.737862 7737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a2f9fafc2643cfbc3094e29f2b24e2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37929 } }
I20260928 06:23:52.739058 7739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97a2f9fafc2643cfbc3094e29f2b24e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a2f9fafc2643cfbc3094e29f2b24e2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37929 } } }
I20260928 06:23:52.739158 7739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:52.739601 7738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97a2f9fafc2643cfbc3094e29f2b24e2. Latest consensus state: current_term: 1 leader_uuid: "97a2f9fafc2643cfbc3094e29f2b24e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a2f9fafc2643cfbc3094e29f2b24e2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 37929 } } }
Sep 28 06:23:52 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:52.740020 7747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:52.740100 7738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:52.740833 7747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:52.743347 7747 catalog_manager.cc:1347] Generated new cluster ID: e617dba16c33434fb5cfa82c121ebb8b
I20260928 06:23:52.743474 7747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:52.747321 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:52.732181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.31.214.126:37929)
0928 06:23:52.732405 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:52.732414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:52.732464 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:23:52.732921 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:52.732930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:52.733001 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:52.733133 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:52.734110 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.735146 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.736342 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.736843 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.736994 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:52.740452 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:52.740457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:52.740457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:52.740471 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:52.742766 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:52.743255 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:52.743258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:52.743258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:52.743397 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:52.746209 (+ 2812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:52.746211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:52.746211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:52.746601 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:52.746813 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:52.746920 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:23:52.747094 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
I20260928 06:23:52.759299 7747 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:52.760011 7747 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:52.775028 7747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2: Generated new TSK 0
I20260928 06:23:52.775310 7747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab.
Entry for principal oryx/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab.
I20260928 06:23:52.806181 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:37929
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab
--principal=oryx/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:52.894307 7758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:52.894522 7758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:52.894547 7758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:52.895751 7758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:52.895802 7758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:52.895823 7758 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:23:52.895841 7758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:52.897452 7758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab
--principal=oryx/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.214.126:37929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:52.897707 7758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:52.897981 7758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:52.903235 7764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:52.903292 7766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:52.903512 7758 server_base.cc:1034] running on GCE node
W20260928 06:23:52.903241 7763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:52.903810 7758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:52.904083 7758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:52.905247 7758 hybrid_clock.cc:648] HybridClock initialized: now 1790576632905221 us; error 40 us; skew 500 ppm
Sep 28 06:23:52 dist-test-slave-q03f krb5kdc[7641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576632, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:52.908349 7758 init.cc:377] Logged in from keytab as oryx/127.31.214.65@KRBTEST.COM (short username oryx)
I20260928 06:23:52.908929 7758 webserver.cc:460] Webserver started at http://127.31.214.65:37175/ using document root <none> and password file <none>
I20260928 06:23:52.909240 7758 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:52.909296 7758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:52.909458 7758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:52.910776 7758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "07eca35f332a45d9a407f793345ee8ab"
format_stamp: "Formatted at 2026-09-28 06:23:52 on dist-test-slave-q03f"
server_key: "584aabe5a0c32a773f8ec68bd1c1f276"
server_key_iv: "a22893b2789599f834ad1e0ae828b53b"
server_key_version: "encryptionkey@0"
I20260928 06:23:52.911195 7758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "07eca35f332a45d9a407f793345ee8ab"
format_stamp: "Formatted at 2026-09-28 06:23:52 on dist-test-slave-q03f"
server_key: "584aabe5a0c32a773f8ec68bd1c1f276"
server_key_iv: "a22893b2789599f834ad1e0ae828b53b"
server_key_version: "encryptionkey@0"
I20260928 06:23:52.912803 7758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:23:52.913694 7773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:52.914078 7758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:52.914196 7758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "07eca35f332a45d9a407f793345ee8ab"
format_stamp: "Formatted at 2026-09-28 06:23:52 on dist-test-slave-q03f"
server_key: "584aabe5a0c32a773f8ec68bd1c1f276"
server_key_iv: "a22893b2789599f834ad1e0ae828b53b"
server_key_version: "encryptionkey@0"
I20260928 06:23:52.914291 7758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:52.944721 7758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:52.947089 7758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:52.947283 7758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:52.947893 7758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:52.947973 7758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:52.948011 7758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:52.948041 7758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:23:52 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576632, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:52.956373 7758 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:41123
I20260928 06:23:52.956481 7888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:41123 every 8 connection(s)
I20260928 06:23:52.956948 7758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:23:52.959091 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:52.948434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:48731 (local address 127.31.214.126:37929)
0928 06:23:52.948527 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:52.948530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:52.948803 (+ 273us) server_negotiation.cc:407] Connection header received
0928 06:23:52.949661 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:52.949662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:52.949674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:52.949765 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:52.950858 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.951384 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.952555 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.952790 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.952871 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:52.955734 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:52.955735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:52.955736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:52.955745 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:52.957614 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:52.958147 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:52.958148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:52.958148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:52.958233 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:52.958465 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:52.958465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:52.958466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:52.958600 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:52.958654 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:52.958919 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 06:23:52.959020 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:23:52.959220 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:52.948399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37929 (local address 127.31.214.65:48731)
0928 06:23:52.948633 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:52.948689 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:52.949502 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:52.949864 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:52.949870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:52.950361 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:52.950738 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:52.950746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.951511 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.951512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:52.952453 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:52.952455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.952879 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.952880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:52.953034 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:52.953605 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:52.953614 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:52.955617 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:52.957700 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:52.957706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:52.957708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:52.958063 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:52.958302 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:52.958302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:52.958303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:52.958408 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:52.958655 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:52.958659 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:52.958782 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 06:23:52.958860 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:23:52.959008 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
I20260928 06:23:52.960047 7889 heartbeater.cc:347] Connected to a master server at 127.31.214.126:37929
I20260928 06:23:52.960155 7889 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:52.960391 7889 heartbeater.cc:511] Master 127.31.214.126:37929 requested a full tablet report, sending...
I20260928 06:23:52.961064 7685 ts_manager.cc:194] Registered new tserver with Master: 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123)
I20260928 06:23:52.961719 7685 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:48731
I20260928 06:23:52.962076 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 7758
I20260928 06:23:52.962198 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:23:52.962424 32601 external_mini_cluster.cc:1428] Setting key 726081cf8ae9005d15a4eca1fbebd85c
I20260928 06:23:52.967868 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:52.961086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:51485 (local address 127.31.214.126:37929)
0928 06:23:52.961127 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:52.961129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:52.962057 (+ 928us) server_negotiation.cc:407] Connection header received
0928 06:23:52.962118 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:52.962119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:52.962129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:52.962176 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:52.962607 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.963145 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.963844 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.964055 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.964142 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:52.965117 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:52.965118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:52.965118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:52.965125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:52.966627 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:52.966997 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:52.966998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:52.966999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:52.967060 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:52.967277 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:52.967278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:52.967279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:52.967428 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:52.967503 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:52.967680 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 06:23:52.967758 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:23:52.967908 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:52.961006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37929 (local address 127.31.214.65:51485)
0928 06:23:52.962000 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:52.962013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:52.962071 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:52.962224 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:52.962225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:52.962446 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:52.962512 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:52.962514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.963239 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.963239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:52.963748 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:52.963749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:52.964143 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:52.964144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:52.964211 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:52.964709 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:52.964714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:52.965014 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:52.966732 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:52.966733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:52.966734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:52.966910 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:52.967137 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:52.967138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:52.967139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:52.967204 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:52.967506 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:52.967507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:52.967556 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:23:52.967635 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:23:52.967757 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab.
Entry for principal oryx/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab.
I20260928 06:23:52.990111 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:37929
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab
--principal=oryx/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:53.074486 7898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:53.074730 7898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:53.074774 7898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:53.076238 7898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:53.076321 7898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:53.076346 7898 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:23:53.076365 7898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:53.078042 7898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab
--principal=oryx/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.214.126:37929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:53.078317 7898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:53.078598 7898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:53.084340 7904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:53.084359 7898 server_base.cc:1034] running on GCE node
W20260928 06:23:53.084369 7903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:53.084342 7906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:53.084770 7898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:53.085054 7898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:53.086196 7898 hybrid_clock.cc:648] HybridClock initialized: now 1790576633086188 us; error 29 us; skew 500 ppm
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576633, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:53.088848 7898 init.cc:377] Logged in from keytab as oryx/127.31.214.66@KRBTEST.COM (short username oryx)
I20260928 06:23:53.089310 7898 webserver.cc:460] Webserver started at http://127.31.214.66:37749/ using document root <none> and password file <none>
I20260928 06:23:53.089566 7898 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:53.089637 7898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:53.089789 7898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:53.090936 7898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "7441bf06a3f94e61a4107e0bf48ae9f8"
format_stamp: "Formatted at 2026-09-28 06:23:53 on dist-test-slave-q03f"
server_key: "1e86c649cf75ed4199f865e8b3bbc71f"
server_key_iv: "31f235113bd23bee6c726f445063a76e"
server_key_version: "encryptionkey@0"
I20260928 06:23:53.091362 7898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "7441bf06a3f94e61a4107e0bf48ae9f8"
format_stamp: "Formatted at 2026-09-28 06:23:53 on dist-test-slave-q03f"
server_key: "1e86c649cf75ed4199f865e8b3bbc71f"
server_key_iv: "31f235113bd23bee6c726f445063a76e"
server_key_version: "encryptionkey@0"
I20260928 06:23:53.092850 7898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:53.093698 7913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:53.093884 7898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:53.094060 7898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "7441bf06a3f94e61a4107e0bf48ae9f8"
format_stamp: "Formatted at 2026-09-28 06:23:53 on dist-test-slave-q03f"
server_key: "1e86c649cf75ed4199f865e8b3bbc71f"
server_key_iv: "31f235113bd23bee6c726f445063a76e"
server_key_version: "encryptionkey@0"
I20260928 06:23:53.094149 7898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:53.115063 7898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:53.117794 7898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:53.118027 7898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:53.118664 7898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:53.118723 7898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:53.118757 7898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:53.118785 7898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576633, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:53.126649 7898 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:33415
I20260928 06:23:53.126690 8028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:33415 every 8 connection(s)
I20260928 06:23:53.127092 7898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:23:53.130162 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.119219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:53269 (local address 127.31.214.126:37929)
0928 06:23:53.119296 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.119299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.119565 (+ 266us) server_negotiation.cc:407] Connection header received
0928 06:23:53.120444 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.120446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.120462 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.120540 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:53.121623 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.122540 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.123663 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.124012 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.124099 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.126988 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:53.126990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:53.126990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:53.126999 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:53.128641 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.129169 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.129170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.129170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.129227 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.129482 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.129483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.129483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.129598 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:53.129652 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.129901 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.130098 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:23:53.130331 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.119187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37929 (local address 127.31.214.66:53269)
0928 06:23:53.119403 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.119455 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.120246 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.120643 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.120648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.121146 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:53.121475 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.121482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.122663 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.122664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.123550 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.123552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.124124 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.124125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.124296 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.124912 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:53.124922 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:53.126881 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:53.128830 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.128838 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.128840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.129080 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.129310 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.129311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.129311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.129416 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.129661 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:53.129665 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:53.129786 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.129877 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.130106 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":108,"threads_started":1}
I20260928 06:23:53.130805 8029 heartbeater.cc:347] Connected to a master server at 127.31.214.126:37929
I20260928 06:23:53.130899 8029 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:53.131096 8029 heartbeater.cc:511] Master 127.31.214.126:37929 requested a full tablet report, sending...
I20260928 06:23:53.131557 7686 ts_manager.cc:194] Registered new tserver with Master: 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415)
I20260928 06:23:53.132066 7686 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:53269
I20260928 06:23:53.135736 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 7898
I20260928 06:23:53.135829 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:53.136019 32601 external_mini_cluster.cc:1428] Setting key 34acec63e55fc76bb3d24fc29991ed35
I20260928 06:23:53.138924 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.132069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:38739 (local address 127.31.214.126:37929)
0928 06:23:53.132119 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.132121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.132230 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:23:53.132319 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.132320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.132328 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.132374 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:53.132740 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.133254 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.134071 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.134248 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.134301 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.135267 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:53.135268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:53.135268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:53.135274 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:53.137601 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.138000 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.138002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.138002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.138062 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.138254 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.138255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.138255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.138465 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:53.138555 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.138796 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.138864 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:23:53.139017 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.131934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37929 (local address 127.31.214.66:38739)
0928 06:23:53.132163 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.132176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.132245 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.132382 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.132382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.132582 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:53.132649 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.132650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.133353 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.133354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.133914 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.133917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.134315 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.134316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.134374 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.134889 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:53.134895 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:53.135182 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:53.137674 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.137675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.137675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.137888 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.138130 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.138131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.138131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.138189 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.138557 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:53.138558 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:53.138615 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.138749 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.138876 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab.
Entry for principal oryx/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab.
I20260928 06:23:53.166218 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:37929
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab
--principal=oryx/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:53.250368 8038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:53.250607 8038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:53.250648 8038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:53.251855 8038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:53.251931 8038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:53.251964 8038 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:23:53.252038 8038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:53.253616 8038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab
--principal=oryx/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.214.126:37929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:53.253909 8038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:53.254200 8038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:53.259485 8046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:53.259487 8043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:53.259486 8044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:53.259756 8038 server_base.cc:1034] running on GCE node
I20260928 06:23:53.260103 8038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:53.260331 8038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:53.261554 8038 hybrid_clock.cc:648] HybridClock initialized: now 1790576633261541 us; error 30 us; skew 500 ppm
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576633, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:53.264629 8038 init.cc:377] Logged in from keytab as oryx/127.31.214.67@KRBTEST.COM (short username oryx)
I20260928 06:23:53.265154 8038 webserver.cc:460] Webserver started at http://127.31.214.67:38885/ using document root <none> and password file <none>
I20260928 06:23:53.265403 8038 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:53.265475 8038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:53.265635 8038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:53.266809 8038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "905d452b01014e4aa80567ef7bec7e46"
format_stamp: "Formatted at 2026-09-28 06:23:53 on dist-test-slave-q03f"
server_key: "989e82f2da09c2a66504b611091a9b41"
server_key_iv: "c66d99f662ea7daca7edb92898ffecfb"
server_key_version: "encryptionkey@0"
I20260928 06:23:53.267197 8038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "905d452b01014e4aa80567ef7bec7e46"
format_stamp: "Formatted at 2026-09-28 06:23:53 on dist-test-slave-q03f"
server_key: "989e82f2da09c2a66504b611091a9b41"
server_key_iv: "c66d99f662ea7daca7edb92898ffecfb"
server_key_version: "encryptionkey@0"
I20260928 06:23:53.268646 8038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:23:53.269490 8053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:53.269687 8038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:53.269800 8038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "905d452b01014e4aa80567ef7bec7e46"
format_stamp: "Formatted at 2026-09-28 06:23:53 on dist-test-slave-q03f"
server_key: "989e82f2da09c2a66504b611091a9b41"
server_key_iv: "c66d99f662ea7daca7edb92898ffecfb"
server_key_version: "encryptionkey@0"
I20260928 06:23:53.269881 8038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:53.280114 8038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:53.282392 8038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:53.282512 8038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:53.283053 8038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:53.283119 8038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:53.283159 8038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:53.283186 8038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576633, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:53.291241 8038 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:36715
I20260928 06:23:53.291649 8038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:53.291616 8168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:36715 every 8 connection(s)
I20260928 06:23:53.293792 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.283473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:47247 (local address 127.31.214.126:37929)
0928 06:23:53.283550 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.283552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.283726 (+ 174us) server_negotiation.cc:407] Connection header received
0928 06:23:53.284263 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.284264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.284280 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.284364 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:53.284991 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.285864 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.286980 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.287334 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.287412 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.290266 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:53.290267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:53.290268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:53.290276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:53.292232 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.292741 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.292743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.292743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.292810 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.293089 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.293090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.293090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.293228 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:53.293306 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.293606 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.293696 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:23:53.294029 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.283441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37929 (local address 127.31.214.67:47247)
0928 06:23:53.283589 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.283630 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.284152 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.284458 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.284462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.284714 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:53.284905 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.284909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.286008 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.286009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.286906 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.286909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.287446 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.287447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.287596 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.288266 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:53.288278 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:53.290156 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:53.292375 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.292382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.292385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.292654 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.292886 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.292887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.292887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.293017 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.293296 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:53.293303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:53.293459 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.293583 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.293755 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260928 06:23:53.294571 8169 heartbeater.cc:347] Connected to a master server at 127.31.214.126:37929
I20260928 06:23:53.294667 8169 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:53.294870 8169 heartbeater.cc:511] Master 127.31.214.126:37929 requested a full tablet report, sending...
I20260928 06:23:53.295323 7685 ts_manager.cc:194] Registered new tserver with Master: 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715)
I20260928 06:23:53.295868 7685 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:47247
I20260928 06:23:53.301330 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 8038
I20260928 06:23:53.301446 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:53.301656 32601 external_mini_cluster.cc:1428] Setting key b2b4a8d8f023e88c4f2e9c3b2330b16b
I20260928 06:23:53.302701 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:23:53.302994 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.295758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:52411 (local address 127.31.214.126:37929)
0928 06:23:53.295800 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.295802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.295867 (+ 65us) server_negotiation.cc:407] Connection header received
0928 06:23:53.295957 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.295958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.295968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.296027 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:53.296322 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.297042 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.297728 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.298014 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.298086 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.299728 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:53.299730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:53.299730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:53.299739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:53.301534 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.302020 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.302021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.302021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.302101 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.302359 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.302361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.302361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.302513 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:53.302607 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.302798 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.302877 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:23:53.303001 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.295667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37929 (local address 127.31.214.67:52411)
0928 06:23:53.295806 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.295817 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.295881 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.296033 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.296034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.296206 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:53.296258 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.296259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.297124 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.297125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.297649 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.297651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.298107 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.298108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.298168 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.298574 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:53.298579 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:53.299599 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:53.301627 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.301628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.301628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.301901 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.302173 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.302174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.302175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.302250 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.302617 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:53.302618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:53.302675 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.302751 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.302893 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:23:53.325594 7735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:53.319592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.31.214.126:37929)
0928 06:23:53.319669 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.319671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.319683 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:23:53.319758 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.319759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.319773 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.319835 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:53.320495 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.321016 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.321889 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.322154 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.322282 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.325501 (+ 3219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40540: BlockingRecv error: recv got EOF from 127.0.0.1:40540 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:53.334296 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.326474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.31.214.126:37929)
0928 06:23:53.326530 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.326532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.326647 (+ 115us) server_negotiation.cc:407] Connection header received
0928 06:23:53.326705 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.326705 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.326714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.326756 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:53.327239 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.327737 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.328418 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.328627 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.328686 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.331229 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:53.331231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:53.331231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:53.331239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:53.332953 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.333400 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.333401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.333402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.333469 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.333693 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.333694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.333694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.333821 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:53.333940 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.334087 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.334173 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:23:53.336184 7685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:53.337905 7685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:53.345925 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.340927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33415 (local address 127.0.0.1:35354)
0928 06:23:53.341288 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.341299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.341361 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.341604 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.341607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.341610 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:53.341683 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.341684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.343175 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.343176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.344409 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.344412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.345546 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.345548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.345641 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.345657 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.345739 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.345797 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":7,"thread_start_us":85,"threads_started":1}
I20260928 06:23:53.345971 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.341109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:41123 (local address 127.0.0.1:39420)
0928 06:23:53.341533 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.341544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.341601 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.342027 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.342028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.342030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:53.342064 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.342065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.342945 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.342946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.344409 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.344412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.345622 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.345623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.345700 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.345706 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.345759 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.345819 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":31,"thread_start_us":65,"threads_started":1}
I20260928 06:23:53.346184 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.341333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.31.214.65:41123)
0928 06:23:53.341779 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.341785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.341794 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:23:53.341840 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.341842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.341856 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.341953 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:53.342156 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.342853 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.344527 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.345398 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.345497 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.345581 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.346013 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.346063 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":83,"threads_started":1}
I20260928 06:23:53.346443 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.340687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:36715 (local address 127.0.0.1:51996)
0928 06:23:53.341034 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.341081 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.341160 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.341642 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.341642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.341645 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:53.341709 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.341712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.343095 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.343096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.345155 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.345157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.346223 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.346224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.346293 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.346297 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.346343 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.346381 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":1365,"thread_start_us":79,"threads_started":1}
I20260928 06:23:53.346592 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.341094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.31.214.66:33415)
0928 06:23:53.341254 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.341259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.341441 (+ 182us) server_negotiation.cc:407] Connection header received
0928 06:23:53.341485 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.341487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.341504 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.341584 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:53.341832 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.342999 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.344537 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.345397 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.345492 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.345895 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.345926 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.346402 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260928 06:23:53.346750 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.340818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.31.214.67:36715)
0928 06:23:53.341025 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.341031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.341156 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:23:53.341459 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.341463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.341482 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.341539 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:53.341822 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.342999 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.345254 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.346022 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.346095 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.346183 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.346593 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.346634 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260928 06:23:53.346956 7823 tablet_service.cc:1467] Processing CreateTablet for tablet a4828c79092842288a579f26de39bb17 (DEFAULT_TABLE table=test-table [id=466a6a78800847c7acca616d4746c75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.347335 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4828c79092842288a579f26de39bb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.347366 8103 tablet_service.cc:1467] Processing CreateTablet for tablet a4828c79092842288a579f26de39bb17 (DEFAULT_TABLE table=test-table [id=466a6a78800847c7acca616d4746c75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.347749 8103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4828c79092842288a579f26de39bb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.350466 7963 tablet_service.cc:1467] Processing CreateTablet for tablet a4828c79092842288a579f26de39bb17 (DEFAULT_TABLE table=test-table [id=466a6a78800847c7acca616d4746c75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.350806 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4828c79092842288a579f26de39bb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.351047 8198 tablet_bootstrap.cc:492] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: Bootstrap starting.
I20260928 06:23:53.351661 8198 tablet_bootstrap.cc:654] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.352072 8198 log.cc:824] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:53.352423 8197 tablet_bootstrap.cc:492] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: Bootstrap starting.
I20260928 06:23:53.352998 8197 tablet_bootstrap.cc:654] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.353251 8197 log.cc:824] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:53.353822 8198 tablet_bootstrap.cc:492] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: No bootstrap required, opened a new log
I20260928 06:23:53.353883 8198 ts_tablet_manager.cc:1397] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:23:53.354324 8201 tablet_bootstrap.cc:492] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: Bootstrap starting.
I20260928 06:23:53.354373 8197 tablet_bootstrap.cc:492] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: No bootstrap required, opened a new log
I20260928 06:23:53.354444 8197 ts_tablet_manager.cc:1397] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:23:53.354852 8201 tablet_bootstrap.cc:654] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.355149 8201 log.cc:824] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:53.355842 8201 tablet_bootstrap.cc:492] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: No bootstrap required, opened a new log
I20260928 06:23:53.355937 8201 ts_tablet_manager.cc:1397] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:53.356383 8198 raft_consensus.cc:348] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.356580 8198 raft_consensus.cc:374] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.356630 8198 raft_consensus.cc:729] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905d452b01014e4aa80567ef7bec7e46, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.356650 8197 raft_consensus.cc:348] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.356810 8197 raft_consensus.cc:374] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.356763 8198 consensus_queue.cc:260] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.356859 8197 raft_consensus.cc:729] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07eca35f332a45d9a407f793345ee8ab, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.357007 8197 consensus_queue.cc:260] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.357101 8169 heartbeater.cc:503] Master 127.31.214.126:37929 was elected leader, sending a full tablet report...
I20260928 06:23:53.357292 8197 ts_tablet_manager.cc:1428] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:23:53.357319 7889 heartbeater.cc:503] Master 127.31.214.126:37929 was elected leader, sending a full tablet report...
I20260928 06:23:53.357455 8198 ts_tablet_manager.cc:1428] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 06:23:53.358016 8201 raft_consensus.cc:348] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.358196 8201 raft_consensus.cc:374] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.358259 8201 raft_consensus.cc:729] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7441bf06a3f94e61a4107e0bf48ae9f8, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.358385 8201 consensus_queue.cc:260] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.358652 8201 ts_tablet_manager.cc:1428] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:23:53.358748 8029 heartbeater.cc:503] Master 127.31.214.126:37929 was elected leader, sending a full tablet report...
W20260928 06:23:53.377728 8030 tablet.cc:2369] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:23:53.379127 8203 raft_consensus.cc:482] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:53.379230 8203 raft_consensus.cc:504] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.379637 8203 leader_election.cc:288] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415), 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123)
I20260928 06:23:53.383272 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.380013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:53731 (local address 127.31.214.66:33415)
0928 06:23:53.380093 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.380095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.380108 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:23:53.380141 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.380142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.380154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.380246 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:53.380562 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.381179 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.382188 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.382771 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.382852 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.382862 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.383157 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.383206 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:23:53.383272 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.380106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:43423 (local address 127.31.214.65:41123)
0928 06:23:53.380155 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.380157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.380283 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:23:53.380353 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.380354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.380364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.380421 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:53.380728 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.381204 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.382334 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.382886 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.382951 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.382957 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.383157 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.383198 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:23:53.383297 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.379833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33415 (local address 127.31.214.67:53731)
0928 06:23:53.379906 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.379920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.379966 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.380250 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.380250 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.380271 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:53.380348 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.380351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.381267 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.381268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.382098 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.382100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.382951 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.382953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.383065 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.383067 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.383134 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.383182 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":4}
I20260928 06:23:53.383453 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.379989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:41123 (local address 127.31.214.67:43423)
0928 06:23:53.380197 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.380210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.380279 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.380563 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.380563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.380565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:53.380654 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.380656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.381328 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.381329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.382251 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.382253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.382972 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.382973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.383060 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.383065 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.383134 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.383271 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260928 06:23:53.383579 8205 raft_consensus.cc:482] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:53.383697 8205 raft_consensus.cc:504] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.383754 7843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4828c79092842288a579f26de39bb17" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07eca35f332a45d9a407f793345ee8ab" is_pre_election: true
I20260928 06:23:53.383940 7843 raft_consensus.cc:2454] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 0.
I20260928 06:23:53.384100 8205 leader_election.cc:288] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715), 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123)
I20260928 06:23:53.384236 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4828c79092842288a579f26de39bb17" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" is_pre_election: true
I20260928 06:23:53.384271 8055 leader_election.cc:302] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07eca35f332a45d9a407f793345ee8ab, 905d452b01014e4aa80567ef7bec7e46; no voters:
I20260928 06:23:53.384342 7983 raft_consensus.cc:2454] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 0.
I20260928 06:23:53.384615 8203 raft_consensus.cc:2781] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:53.384723 8203 raft_consensus.cc:482] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:53.384774 8203 raft_consensus.cc:3037] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.386117 8203 raft_consensus.cc:504] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.386267 8203 leader_election.cc:288] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415), 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123)
I20260928 06:23:53.386536 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4828c79092842288a579f26de39bb17" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8"
I20260928 06:23:53.386612 7983 raft_consensus.cc:3037] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.386607 7843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4828c79092842288a579f26de39bb17" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07eca35f332a45d9a407f793345ee8ab"
I20260928 06:23:53.386677 7843 raft_consensus.cc:3037] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.387535 7983 raft_consensus.cc:2454] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 1.
I20260928 06:23:53.387651 7843 raft_consensus.cc:2454] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 1.
I20260928 06:23:53.387712 8055 leader_election.cc:302] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7441bf06a3f94e61a4107e0bf48ae9f8, 905d452b01014e4aa80567ef7bec7e46; no voters:
I20260928 06:23:53.388028 8203 raft_consensus.cc:2781] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:53.388222 8203 raft_consensus.cc:686] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 905d452b01014e4aa80567ef7bec7e46, State: Running, Role: LEADER
I20260928 06:23:53.388329 8203 consensus_queue.cc:237] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } }
I20260928 06:23:53.388491 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.384222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:36715 (local address 127.31.214.66:54825)
0928 06:23:53.384439 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.384452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.384499 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.384850 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.384851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.384853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:53.384916 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.384918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.385833 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.385834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.387307 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.387309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.388174 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.388175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.388266 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.388270 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.388325 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.388381 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260928 06:23:53.388574 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.384409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:59701 (local address 127.31.214.65:41123)
0928 06:23:53.384538 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.384554 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.384690 (+ 136us) server_negotiation.cc:407] Connection header received
0928 06:23:53.384774 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.384774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.384784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.384812 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:53.385014 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.385436 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.387591 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.388192 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.388234 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.388244 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.388460 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.388500 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 06:23:53.388692 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.384631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:54825 (local address 127.31.214.67:36715)
0928 06:23:53.384698 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.384700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.384709 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:23:53.384731 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.384732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.384743 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.384803 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:53.384999 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.385729 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.387465 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.388061 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.388131 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.388139 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.388590 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.388634 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:23:53.388784 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.384323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:41123 (local address 127.31.214.66:59701)
0928 06:23:53.384631 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.384642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.384700 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.384870 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.384870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.384872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:53.384944 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.384945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.385522 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.385523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.387496 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.387498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.388304 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.388305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.388374 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.388377 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.388642 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.388682 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":104,"threads_started":1}
I20260928 06:23:53.389045 7843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4828c79092842288a579f26de39bb17" candidate_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07eca35f332a45d9a407f793345ee8ab" is_pre_election: true
I20260928 06:23:53.389143 7843 raft_consensus.cc:2379] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7441bf06a3f94e61a4107e0bf48ae9f8 in current term 1: Already voted for candidate 905d452b01014e4aa80567ef7bec7e46 in this term.
I20260928 06:23:53.389362 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4828c79092842288a579f26de39bb17" candidate_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905d452b01014e4aa80567ef7bec7e46" is_pre_election: true
I20260928 06:23:53.389364 7685 catalog_manager.cc:5495] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67). New cstate: current_term: 1 leader_uuid: "905d452b01014e4aa80567ef7bec7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:53.389634 7915 leader_election.cc:302] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7441bf06a3f94e61a4107e0bf48ae9f8; no voters: 07eca35f332a45d9a407f793345ee8ab, 905d452b01014e4aa80567ef7bec7e46
I20260928 06:23:53.389824 8205 raft_consensus.cc:2726] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 06:23:53.404536 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.402041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.31.214.67:36715)
0928 06:23:53.402113 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.402115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.402125 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:23:53.402164 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.402164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.402181 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.402262 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:53.402406 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.402901 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.403682 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.403879 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.403955 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.404162 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:53.404279 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:53.404381 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.404401 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.404420 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:23:53.406829 7983 raft_consensus.cc:1261] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Refusing update from remote peer 905d452b01014e4aa80567ef7bec7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.406863 7843 raft_consensus.cc:1261] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Refusing update from remote peer 905d452b01014e4aa80567ef7bec7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.407230 8208 consensus_queue.cc:1035] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.407394 8203 consensus_queue.cc:1035] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.410120 8211 mvcc.cc:205] Tried to move back new op lower bound from 7334201890430881792 to 7334201890359324672. Current Snapshot: MvccSnapshot[applied={T|T < 7334201890430881792}]
I20260928 06:23:53.411165 8212 mvcc.cc:205] Tried to move back new op lower bound from 7334201890430881792 to 7334201890359324672. Current Snapshot: MvccSnapshot[applied={T|T < 7334201890430881792}]
I20260928 06:23:53.412599 7685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:53.412694 7685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:53.413496 7685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2: Sending DeleteTablet for 3 replicas of tablet a4828c79092842288a579f26de39bb17
I20260928 06:23:53.413915 7823 tablet_service.cc:1514] Processing DeleteTablet for tablet a4828c79092842288a579f26de39bb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:39420
I20260928 06:23:53.413918 7963 tablet_service.cc:1514] Processing DeleteTablet for tablet a4828c79092842288a579f26de39bb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:35354
I20260928 06:23:53.414193 8103 tablet_service.cc:1514] Processing DeleteTablet for tablet a4828c79092842288a579f26de39bb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:51996
I20260928 06:23:53.414165 7685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:53.414307 7685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:53.414431 8223 tablet_replica.cc:330] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: stopping tablet replica
I20260928 06:23:53.414538 8223 raft_consensus.cc:2229] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:23:53.414602 8222 tablet_replica.cc:330] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: stopping tablet replica
I20260928 06:23:53.414628 8223 raft_consensus.cc:2258] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.414700 8222 raft_consensus.cc:2229] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:53.414793 8222 raft_consensus.cc:2258] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.415058 8223 ts_tablet_manager.cc:1905] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.415239 8221 tablet_replica.cc:330] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: stopping tablet replica
I20260928 06:23:53.415331 8221 raft_consensus.cc:2229] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:53.415441 8221 raft_consensus.cc:2258] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.415894 8221 ts_tablet_manager.cc:1905] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.416196 7963 tablet_service.cc:1467] Processing CreateTablet for tablet c2ddb8a3498b4add946a1c66f87765d5 (DEFAULT_TABLE table=test-table [id=e5bbd3930c57405998385ca5bebb18ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.416249 7823 tablet_service.cc:1467] Processing CreateTablet for tablet c2ddb8a3498b4add946a1c66f87765d5 (DEFAULT_TABLE table=test-table [id=e5bbd3930c57405998385ca5bebb18ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.416304 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ddb8a3498b4add946a1c66f87765d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.416335 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ddb8a3498b4add946a1c66f87765d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.416848 8223 ts_tablet_manager.cc:1918] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:53.416929 8223 log.cc:1196] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/a4828c79092842288a579f26de39bb17
I20260928 06:23:53.417268 8223 ts_tablet_manager.cc:1939] T a4828c79092842288a579f26de39bb17 P 905d452b01014e4aa80567ef7bec7e46: Deleting consensus metadata
I20260928 06:23:53.417655 8201 tablet_bootstrap.cc:492] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: Bootstrap starting.
I20260928 06:23:53.417686 8197 tablet_bootstrap.cc:492] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: Bootstrap starting.
I20260928 06:23:53.417851 8103 tablet_service.cc:1467] Processing CreateTablet for tablet c2ddb8a3498b4add946a1c66f87765d5 (DEFAULT_TABLE table=test-table [id=e5bbd3930c57405998385ca5bebb18ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.417981 8103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ddb8a3498b4add946a1c66f87765d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.418102 8201 tablet_bootstrap.cc:654] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.418083 7671 catalog_manager.cc:4841] TS 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715): tablet a4828c79092842288a579f26de39bb17 (table test-table [id=466a6a78800847c7acca616d4746c75d]) successfully deleted
I20260928 06:23:53.418538 8197 tablet_bootstrap.cc:654] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.419287 8198 tablet_bootstrap.cc:492] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: Bootstrap starting.
I20260928 06:23:53.419366 8197 tablet_bootstrap.cc:492] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: No bootstrap required, opened a new log
I20260928 06:23:53.419421 8197 ts_tablet_manager.cc:1397] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:53.419595 8197 raft_consensus.cc:348] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.419695 8197 raft_consensus.cc:374] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.419723 8198 tablet_bootstrap.cc:654] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.419730 8197 raft_consensus.cc:729] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07eca35f332a45d9a407f793345ee8ab, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.419782 8197 consensus_queue.cc:260] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.419819 8221 ts_tablet_manager.cc:1918] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:53.419902 8197 ts_tablet_manager.cc:1428] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:53.420169 8222 ts_tablet_manager.cc:1905] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.420531 8198 tablet_bootstrap.cc:492] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: No bootstrap required, opened a new log
I20260928 06:23:53.420581 8198 ts_tablet_manager.cc:1397] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:53.420730 8198 raft_consensus.cc:348] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.420794 8198 raft_consensus.cc:374] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.420820 8198 raft_consensus.cc:729] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905d452b01014e4aa80567ef7bec7e46, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.420879 8198 consensus_queue.cc:260] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.420966 8198 ts_tablet_manager.cc:1428] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:53.421715 8222 ts_tablet_manager.cc:1918] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:53.421789 8222 log.cc:1196] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/a4828c79092842288a579f26de39bb17
I20260928 06:23:53.422091 8222 ts_tablet_manager.cc:1939] T a4828c79092842288a579f26de39bb17 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting consensus metadata
I20260928 06:23:53.422349 8201 tablet_bootstrap.cc:492] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: No bootstrap required, opened a new log
I20260928 06:23:53.422412 8201 ts_tablet_manager.cc:1397] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:23:53.422557 8201 raft_consensus.cc:348] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.422632 8201 raft_consensus.cc:374] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.422660 8201 raft_consensus.cc:729] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7441bf06a3f94e61a4107e0bf48ae9f8, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.422709 8201 consensus_queue.cc:260] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.422792 8201 ts_tablet_manager.cc:1428] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:53.419912 8221 log.cc:1196] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/a4828c79092842288a579f26de39bb17
I20260928 06:23:53.423170 7671 catalog_manager.cc:4841] TS 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415): tablet a4828c79092842288a579f26de39bb17 (table test-table [id=466a6a78800847c7acca616d4746c75d]) successfully deleted
I20260928 06:23:53.423406 8221 ts_tablet_manager.cc:1939] T a4828c79092842288a579f26de39bb17 P 07eca35f332a45d9a407f793345ee8ab: Deleting consensus metadata
I20260928 06:23:53.423799 7671 catalog_manager.cc:4841] TS 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123): tablet a4828c79092842288a579f26de39bb17 (table test-table [id=466a6a78800847c7acca616d4746c75d]) successfully deleted
W20260928 06:23:53.457747 7890 tablet.cc:2369] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:23:53.466503 8203 raft_consensus.cc:482] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:53.466660 8203 raft_consensus.cc:504] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.466859 8203 leader_election.cc:288] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123), 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415)
I20260928 06:23:53.467103 7843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ddb8a3498b4add946a1c66f87765d5" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07eca35f332a45d9a407f793345ee8ab" is_pre_election: true
I20260928 06:23:53.467123 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ddb8a3498b4add946a1c66f87765d5" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" is_pre_election: true
I20260928 06:23:53.467204 7843 raft_consensus.cc:2454] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 0.
I20260928 06:23:53.467206 7983 raft_consensus.cc:2454] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 0.
I20260928 06:23:53.467382 8055 leader_election.cc:302] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07eca35f332a45d9a407f793345ee8ab, 905d452b01014e4aa80567ef7bec7e46; no voters:
I20260928 06:23:53.467530 8203 raft_consensus.cc:2781] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:53.467581 8203 raft_consensus.cc:482] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:53.467625 8203 raft_consensus.cc:3037] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.468640 8203 raft_consensus.cc:504] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.468779 8203 leader_election.cc:288] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123), 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415)
I20260928 06:23:53.468966 7843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ddb8a3498b4add946a1c66f87765d5" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07eca35f332a45d9a407f793345ee8ab"
I20260928 06:23:53.469043 7843 raft_consensus.cc:3037] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.469197 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ddb8a3498b4add946a1c66f87765d5" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8"
I20260928 06:23:53.469277 7983 raft_consensus.cc:3037] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.469990 7843 raft_consensus.cc:2454] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 1.
I20260928 06:23:53.470147 7983 raft_consensus.cc:2454] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 1.
I20260928 06:23:53.470170 8055 leader_election.cc:302] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07eca35f332a45d9a407f793345ee8ab, 905d452b01014e4aa80567ef7bec7e46; no voters:
I20260928 06:23:53.470320 8203 raft_consensus.cc:2781] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:53.470384 8203 raft_consensus.cc:686] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 905d452b01014e4aa80567ef7bec7e46, State: Running, Role: LEADER
I20260928 06:23:53.470438 8203 consensus_queue.cc:237] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.471046 7685 catalog_manager.cc:5495] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67). New cstate: current_term: 1 leader_uuid: "905d452b01014e4aa80567ef7bec7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } health_report { overall_health: HEALTHY } } }
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576632, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.126@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:53.511487 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.502782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:37929 (local address 127.0.0.1:40550)
0928 06:23:53.502985 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.502999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.503092 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.503277 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.503278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.503716 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:53.503783 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.503787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.504622 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.504624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.505491 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.505495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.505984 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.505985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.506056 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.506706 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:53.506712 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:53.508395 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:53.510342 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.510349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.510351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.510564 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.510829 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:53.510830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:53.510830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:53.510888 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:53.511210 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:53.511216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:53.511260 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.511309 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.511408 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260928 06:23:53.511633 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.502855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.31.214.126:37929)
0928 06:23:53.502970 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.502978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.503081 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:23:53.503166 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.503166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.503176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.503234 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:53.503951 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.504521 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.505603 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.505866 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.505929 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.508492 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:53.508494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:53.508494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:53.508503 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:53.510247 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.510657 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.510658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.510659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.510749 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.510962 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.510963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.510964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.511133 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:53.511208 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.511350 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.511480 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":84}
I20260928 06:23:53.513913 7685 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:40550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:23:53.514240 7685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:53.516093 8103 tablet_service.cc:1467] Processing CreateTablet for tablet bc20fe402de242e9bc3026fc59c41253 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b7d4889a83b44cdb4d1374727a6e4aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:23:53.516081 7963 tablet_service.cc:1467] Processing CreateTablet for tablet bc20fe402de242e9bc3026fc59c41253 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b7d4889a83b44cdb4d1374727a6e4aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:23:53.516268 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc20fe402de242e9bc3026fc59c41253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.516266 8103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc20fe402de242e9bc3026fc59c41253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.517959 8201 tablet_bootstrap.cc:492] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: Bootstrap starting.
I20260928 06:23:53.518020 7823 tablet_service.cc:1467] Processing CreateTablet for tablet bc20fe402de242e9bc3026fc59c41253 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b7d4889a83b44cdb4d1374727a6e4aa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:23:53.518170 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc20fe402de242e9bc3026fc59c41253. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.518437 8201 tablet_bootstrap.cc:654] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.519356 8201 tablet_bootstrap.cc:492] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: No bootstrap required, opened a new log
I20260928 06:23:53.519424 8201 ts_tablet_manager.cc:1397] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:53.519584 8201 raft_consensus.cc:348] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.519671 8201 raft_consensus.cc:374] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.519697 8201 raft_consensus.cc:729] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7441bf06a3f94e61a4107e0bf48ae9f8, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.519759 8197 tablet_bootstrap.cc:492] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: Bootstrap starting.
I20260928 06:23:53.519761 8201 consensus_queue.cc:260] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.519799 8198 tablet_bootstrap.cc:492] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: Bootstrap starting.
I20260928 06:23:53.519887 8201 ts_tablet_manager.cc:1428] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:53.520025 8205 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.520090 8205 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:53.520218 8197 tablet_bootstrap.cc:654] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.520237 8198 tablet_bootstrap.cc:654] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.520957 8198 tablet_bootstrap.cc:492] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: No bootstrap required, opened a new log
I20260928 06:23:53.521020 8198 ts_tablet_manager.cc:1397] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:53.521195 8198 raft_consensus.cc:348] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.521309 8198 raft_consensus.cc:374] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.521343 8198 raft_consensus.cc:729] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905d452b01014e4aa80567ef7bec7e46, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.521400 8198 consensus_queue.cc:260] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.521534 8198 ts_tablet_manager.cc:1428] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:23:53.521600 8197 tablet_bootstrap.cc:492] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: No bootstrap required, opened a new log
I20260928 06:23:53.521657 8197 ts_tablet_manager.cc:1397] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:23:53.521643 8203 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.521739 8203 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:53.521842 8197 raft_consensus.cc:348] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.521924 8197 raft_consensus.cc:374] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.521986 8197 raft_consensus.cc:729] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07eca35f332a45d9a407f793345ee8ab, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.522032 8197 consensus_queue.cc:260] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.522132 8197 ts_tablet_manager.cc:1428] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:53.522185 8204 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.522254 8204 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:53.527333 8204 raft_consensus.cc:482] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:53.527447 8204 raft_consensus.cc:504] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.527787 8204 leader_election.cc:288] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415), 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715)
I20260928 06:23:53.531627 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.527985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33415 (local address 127.31.214.65:38961)
0928 06:23:53.528185 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.528198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.528279 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.528453 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.528454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.528462 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:53.528565 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.528568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.529486 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.529487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.530721 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.530724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.531325 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.531326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.531411 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.531417 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.531489 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.531532 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":110,"threads_started":1}
I20260928 06:23:53.531627 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.528058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:38961 (local address 127.31.214.66:33415)
0928 06:23:53.528132 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.528135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.528250 (+ 115us) server_negotiation.cc:407] Connection header received
0928 06:23:53.528339 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.528341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.528350 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.528403 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:53.528638 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.529381 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.530797 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.531243 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.531327 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.531338 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.531504 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.531548 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:23:53.531991 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc20fe402de242e9bc3026fc59c41253" candidate_uuid: "07eca35f332a45d9a407f793345ee8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" is_pre_election: true
I20260928 06:23:53.532099 7983 raft_consensus.cc:2454] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07eca35f332a45d9a407f793345ee8ab in term 0.
I20260928 06:23:53.532279 7775 leader_election.cc:302] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07eca35f332a45d9a407f793345ee8ab, 7441bf06a3f94e61a4107e0bf48ae9f8; no voters:
I20260928 06:23:53.532437 8204 raft_consensus.cc:2781] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:53.532475 8204 raft_consensus.cc:482] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:53.532507 8204 raft_consensus.cc:3037] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.532787 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.528336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:33827 (local address 127.31.214.67:36715)
0928 06:23:53.528387 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.528389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.528528 (+ 139us) server_negotiation.cc:407] Connection header received
0928 06:23:53.528561 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.528562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.528571 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.528627 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:23:53.529715 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.530443 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.531831 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.532379 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.532429 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.532437 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.532674 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.532725 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:23:53.532784 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.528215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:36715 (local address 127.31.214.65:33827)
0928 06:23:53.528408 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.528420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.528501 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.529529 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.529531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.529534 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:23:53.529608 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.529610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.530567 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.530567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.531709 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.531712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.532503 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.532504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.532591 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.532595 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.532654 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.532696 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":50,"threads_started":1}
I20260928 06:23:53.533460 8204 raft_consensus.cc:504] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.533599 8204 leader_election.cc:288] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [CANDIDATE]: Term 1 election: Requested vote from peers 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415), 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715)
I20260928 06:23:53.533859 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc20fe402de242e9bc3026fc59c41253" candidate_uuid: "07eca35f332a45d9a407f793345ee8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8"
I20260928 06:23:53.533990 7983 raft_consensus.cc:3037] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.533934 8123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc20fe402de242e9bc3026fc59c41253" candidate_uuid: "07eca35f332a45d9a407f793345ee8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905d452b01014e4aa80567ef7bec7e46" is_pre_election: true
I20260928 06:23:53.534101 8123 raft_consensus.cc:2454] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07eca35f332a45d9a407f793345ee8ab in term 0.
I20260928 06:23:53.533977 8119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc20fe402de242e9bc3026fc59c41253" candidate_uuid: "07eca35f332a45d9a407f793345ee8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905d452b01014e4aa80567ef7bec7e46"
I20260928 06:23:53.534225 8119 raft_consensus.cc:3037] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.534752 7983 raft_consensus.cc:2454] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07eca35f332a45d9a407f793345ee8ab in term 1.
I20260928 06:23:53.534878 8119 raft_consensus.cc:2454] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07eca35f332a45d9a407f793345ee8ab in term 1.
I20260928 06:23:53.534943 7775 leader_election.cc:302] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07eca35f332a45d9a407f793345ee8ab, 7441bf06a3f94e61a4107e0bf48ae9f8; no voters:
I20260928 06:23:53.535127 8204 raft_consensus.cc:2781] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:53.535276 8204 raft_consensus.cc:686] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [term 1 LEADER]: Becoming Leader. State: Replica: 07eca35f332a45d9a407f793345ee8ab, State: Running, Role: LEADER
I20260928 06:23:53.535348 8204 consensus_queue.cc:237] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.535712 8239 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: TxnStatusTablet state changed. Reason: New leader 07eca35f332a45d9a407f793345ee8ab. Latest consensus state: current_term: 1 leader_uuid: "07eca35f332a45d9a407f793345ee8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.535852 8239 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:23:53.536099 8241 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:23:53.536088 7685 catalog_manager.cc:5495] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 07eca35f332a45d9a407f793345ee8ab (127.31.214.65). New cstate: current_term: 1 leader_uuid: "07eca35f332a45d9a407f793345ee8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:53.536238 8241 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260928 06:23:53.542253 8170 tablet.cc:2369] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:23:53.544363 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.541334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.31.214.65:41123)
0928 06:23:53.541396 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.541398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.541438 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:23:53.541537 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.541538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.541558 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.541627 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:53.541799 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.542572 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.543471 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.543701 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.543784 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.544005 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:53.544120 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:53.544205 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.544266 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.544287 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:23:53.544404 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.541297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:41123 (local address 127.0.0.1:39428)
0928 06:23:53.541386 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.541398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.541460 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.541643 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.541644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.541646 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:23:53.541700 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.541702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.542673 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.542674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.543356 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.543359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.543785 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.543786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.543860 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.543891 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:23:53.544204 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:23:53.544215 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.544264 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.544320 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 06:23:53.545768 7983 raft_consensus.cc:1261] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Refusing update from remote peer 07eca35f332a45d9a407f793345ee8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.545804 8119 raft_consensus.cc:1261] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Refusing update from remote peer 07eca35f332a45d9a407f793345ee8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.546201 8204 consensus_queue.cc:1035] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.546351 8239 consensus_queue.cc:1035] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.547082 8203 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: TxnStatusTablet state changed. Reason: New leader 07eca35f332a45d9a407f793345ee8ab. Latest consensus state: current_term: 1 leader_uuid: "07eca35f332a45d9a407f793345ee8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.547089 8205 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: TxnStatusTablet state changed. Reason: New leader 07eca35f332a45d9a407f793345ee8ab. Latest consensus state: current_term: 1 leader_uuid: "07eca35f332a45d9a407f793345ee8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.547191 8203 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:53.547199 8205 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:53.551064 8204 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07eca35f332a45d9a407f793345ee8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.551182 8204 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:23:53.551229 8204 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07eca35f332a45d9a407f793345ee8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.551265 8204 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 07eca35f332a45d9a407f793345ee8ab: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:23:53.551632 8213 mvcc.cc:205] Tried to move back new op lower bound from 7334201891001376768 to 7334201890961424384. Current Snapshot: MvccSnapshot[applied={T|T < 7334201891001376768}]
I20260928 06:23:53.551757 8205 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07eca35f332a45d9a407f793345ee8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.551831 8205 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 7441bf06a3f94e61a4107e0bf48ae9f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:53.552536 8203 tablet_replica.cc:439] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07eca35f332a45d9a407f793345ee8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } } }
I20260928 06:23:53.552649 8203 tablet_replica.cc:442] T bc20fe402de242e9bc3026fc59c41253 P 905d452b01014e4aa80567ef7bec7e46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:23:53.558516 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.555737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:41123 (local address 127.31.214.67:49183)
0928 06:23:53.555829 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.555843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.556024 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.556174 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.556175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.556177 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:23:53.556229 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.556231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.556946 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.556947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.557588 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.557590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.558030 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.558031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.558096 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.558119 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:23:53.558327 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:23:53.558330 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.558382 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.558432 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:23:53.558487 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.555837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:49183 (local address 127.31.214.65:41123)
0928 06:23:53.555909 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.555912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.556013 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:23:53.556105 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.556106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.556118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.556172 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:53.556290 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.556850 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.557685 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.557908 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.558029 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.558211 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:53.558257 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:53.558327 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.558384 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.558406 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:23:53.561084 7843 raft_consensus.cc:1261] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Refusing update from remote peer 905d452b01014e4aa80567ef7bec7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.561389 8218 consensus_queue.cc:1035] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.561511 7983 raft_consensus.cc:1261] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Refusing update from remote peer 905d452b01014e4aa80567ef7bec7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.561751 8218 consensus_queue.cc:1035] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.571434 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.568205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:36715 (local address 127.31.214.65:60293)
0928 06:23:53.568420 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.568433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:53.568504 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:53.568640 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:53.568641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:53.568654 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:23:53.568729 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.568731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.569506 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.569507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.570447 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:53.570451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.570993 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.570994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:53.571049 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.571067 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:23:53.571271 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:23:53.571272 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:23:53.571304 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 06:23:53.571340 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":132,"threads_started":1}
I20260928 06:23:53.571421 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.568272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:60293 (local address 127.31.214.67:36715)
0928 06:23:53.568342 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.568345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.568502 (+ 157us) server_negotiation.cc:407] Connection header received
0928 06:23:53.568570 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.568571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.568581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.568627 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:53.568799 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.569385 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.570539 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.570858 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.570943 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.571168 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:53.571206 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:53.571268 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.571310 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.571334 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:23:53.579972 7686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:53.580109 7686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:53.580880 7686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2: Sending DeleteTablet for 3 replicas of tablet c2ddb8a3498b4add946a1c66f87765d5
I20260928 06:23:53.581183 7963 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ddb8a3498b4add946a1c66f87765d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:35354
I20260928 06:23:53.581183 7823 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ddb8a3498b4add946a1c66f87765d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:39420
I20260928 06:23:53.581207 8103 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ddb8a3498b4add946a1c66f87765d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:51996
I20260928 06:23:53.581279 8221 tablet_replica.cc:330] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: stopping tablet replica
I20260928 06:23:53.581279 8222 tablet_replica.cc:330] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: stopping tablet replica
I20260928 06:23:53.581321 8221 raft_consensus.cc:2229] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:53.581331 8222 raft_consensus.cc:2229] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:53.581326 8223 tablet_replica.cc:330] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: stopping tablet replica
I20260928 06:23:53.581358 8221 raft_consensus.cc:2258] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.581364 8222 raft_consensus.cc:2258] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.581390 8223 raft_consensus.cc:2229] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:23:53.581467 8223 raft_consensus.cc:2258] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.581717 8222 ts_tablet_manager.cc:1905] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.581769 8221 ts_tablet_manager.cc:1905] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.581775 8223 ts_tablet_manager.cc:1905] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.584776 8223 ts_tablet_manager.cc:1918] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:53.584851 8223 log.cc:1196] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/c2ddb8a3498b4add946a1c66f87765d5
I20260928 06:23:53.585089 8223 ts_tablet_manager.cc:1939] T c2ddb8a3498b4add946a1c66f87765d5 P 905d452b01014e4aa80567ef7bec7e46: Deleting consensus metadata
I20260928 06:23:53.585153 8221 ts_tablet_manager.cc:1918] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:53.585220 8221 log.cc:1196] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/c2ddb8a3498b4add946a1c66f87765d5
I20260928 06:23:53.585464 8221 ts_tablet_manager.cc:1939] T c2ddb8a3498b4add946a1c66f87765d5 P 07eca35f332a45d9a407f793345ee8ab: Deleting consensus metadata
I20260928 06:23:53.585444 7671 catalog_manager.cc:4841] TS 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715): tablet c2ddb8a3498b4add946a1c66f87765d5 (table test-table [id=e5bbd3930c57405998385ca5bebb18ba]) successfully deleted
I20260928 06:23:53.586087 7671 catalog_manager.cc:4841] TS 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123): tablet c2ddb8a3498b4add946a1c66f87765d5 (table test-table [id=e5bbd3930c57405998385ca5bebb18ba]) successfully deleted
I20260928 06:23:53.586961 8222 ts_tablet_manager.cc:1918] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:53.587023 8222 log.cc:1196] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/c2ddb8a3498b4add946a1c66f87765d5
I20260928 06:23:53.587260 8222 ts_tablet_manager.cc:1939] T c2ddb8a3498b4add946a1c66f87765d5 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting consensus metadata
I20260928 06:23:53.587596 7671 catalog_manager.cc:4841] TS 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415): tablet c2ddb8a3498b4add946a1c66f87765d5 (table test-table [id=e5bbd3930c57405998385ca5bebb18ba]) successfully deleted
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:23:53.607535 7735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:53.601808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.31.214.126:37929)
0928 06:23:53.601889 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.601891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.602035 (+ 144us) server_negotiation.cc:407] Connection header received
0928 06:23:53.602121 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.602122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.602133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.602187 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:53.602796 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.603353 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.604161 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.604378 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.604449 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.607432 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40554: BlockingRecv error: recv got EOF from 127.0.0.1:40554 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[7641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM
I20260928 06:23:53.616457 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.608453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.31.214.126:37929)
0928 06:23:53.608503 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.608505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.608524 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:23:53.608584 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.608585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.608598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.608683 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:53.609099 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.609858 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.610617 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.610928 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.611009 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.613589 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:53.613591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:53.613591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:53.613601 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:53.615317 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.615661 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.615662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.615662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.615732 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:53.615929 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:53.615930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:53.615931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:53.616075 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:53.616191 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.616268 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.616351 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:23:53.618304 7686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:53.618530 7686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:53.620085 7823 tablet_service.cc:1467] Processing CreateTablet for tablet ede26f1559a745408fe8c98f4aa4b106 (DEFAULT_TABLE table=test-table [id=01a688154af94665ac70ef3a06f3d5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.620149 8103 tablet_service.cc:1467] Processing CreateTablet for tablet ede26f1559a745408fe8c98f4aa4b106 (DEFAULT_TABLE table=test-table [id=01a688154af94665ac70ef3a06f3d5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.620149 7963 tablet_service.cc:1467] Processing CreateTablet for tablet ede26f1559a745408fe8c98f4aa4b106 (DEFAULT_TABLE table=test-table [id=01a688154af94665ac70ef3a06f3d5cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.620246 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede26f1559a745408fe8c98f4aa4b106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.620280 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede26f1559a745408fe8c98f4aa4b106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.620280 8103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede26f1559a745408fe8c98f4aa4b106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.621655 8198 tablet_bootstrap.cc:492] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: Bootstrap starting.
I20260928 06:23:53.621670 8197 tablet_bootstrap.cc:492] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: Bootstrap starting.
I20260928 06:23:53.622057 8198 tablet_bootstrap.cc:654] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.622138 8197 tablet_bootstrap.cc:654] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.622359 8201 tablet_bootstrap.cc:492] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: Bootstrap starting.
I20260928 06:23:53.622773 8198 tablet_bootstrap.cc:492] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: No bootstrap required, opened a new log
I20260928 06:23:53.622787 8201 tablet_bootstrap.cc:654] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.622839 8198 ts_tablet_manager.cc:1397] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:53.622960 8198 raft_consensus.cc:348] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.623041 8198 raft_consensus.cc:374] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.623065 8198 raft_consensus.cc:729] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905d452b01014e4aa80567ef7bec7e46, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.623118 8198 consensus_queue.cc:260] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.623220 8198 ts_tablet_manager.cc:1428] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:23:53.623682 8201 tablet_bootstrap.cc:492] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: No bootstrap required, opened a new log
I20260928 06:23:53.623752 8201 ts_tablet_manager.cc:1397] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:53.623920 8201 raft_consensus.cc:348] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.624022 8201 raft_consensus.cc:374] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.624054 8201 raft_consensus.cc:729] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7441bf06a3f94e61a4107e0bf48ae9f8, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.624105 8201 consensus_queue.cc:260] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.624194 8201 ts_tablet_manager.cc:1428] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:53.624825 8197 tablet_bootstrap.cc:492] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: No bootstrap required, opened a new log
I20260928 06:23:53.624883 8197 ts_tablet_manager.cc:1397] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:23:53.624987 8197 raft_consensus.cc:348] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.625058 8197 raft_consensus.cc:374] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.625080 8197 raft_consensus.cc:729] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07eca35f332a45d9a407f793345ee8ab, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.625134 8197 consensus_queue.cc:260] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.625228 8197 ts_tablet_manager.cc:1428] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:53.756716 8239 raft_consensus.cc:482] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:53.756831 8239 raft_consensus.cc:504] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.757051 8239 leader_election.cc:288] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415), 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715)
I20260928 06:23:53.757366 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede26f1559a745408fe8c98f4aa4b106" candidate_uuid: "07eca35f332a45d9a407f793345ee8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" is_pre_election: true
I20260928 06:23:53.757427 8119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede26f1559a745408fe8c98f4aa4b106" candidate_uuid: "07eca35f332a45d9a407f793345ee8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905d452b01014e4aa80567ef7bec7e46" is_pre_election: true
I20260928 06:23:53.757490 7983 raft_consensus.cc:2454] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07eca35f332a45d9a407f793345ee8ab in term 0.
I20260928 06:23:53.757555 8119 raft_consensus.cc:2454] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07eca35f332a45d9a407f793345ee8ab in term 0.
I20260928 06:23:53.757692 7775 leader_election.cc:302] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07eca35f332a45d9a407f793345ee8ab, 7441bf06a3f94e61a4107e0bf48ae9f8; no voters:
I20260928 06:23:53.757840 8239 raft_consensus.cc:2781] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:53.757899 8239 raft_consensus.cc:482] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:53.757915 8239 raft_consensus.cc:3037] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.758760 8239 raft_consensus.cc:504] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.758888 8239 leader_election.cc:288] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [CANDIDATE]: Term 1 election: Requested vote from peers 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415), 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715)
I20260928 06:23:53.759220 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede26f1559a745408fe8c98f4aa4b106" candidate_uuid: "07eca35f332a45d9a407f793345ee8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8"
I20260928 06:23:53.759346 7983 raft_consensus.cc:3037] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.759328 8119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede26f1559a745408fe8c98f4aa4b106" candidate_uuid: "07eca35f332a45d9a407f793345ee8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "905d452b01014e4aa80567ef7bec7e46"
I20260928 06:23:53.759454 8119 raft_consensus.cc:3037] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.760259 8119 raft_consensus.cc:2454] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07eca35f332a45d9a407f793345ee8ab in term 1.
I20260928 06:23:53.760259 7983 raft_consensus.cc:2454] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07eca35f332a45d9a407f793345ee8ab in term 1.
I20260928 06:23:53.760459 7775 leader_election.cc:302] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07eca35f332a45d9a407f793345ee8ab, 7441bf06a3f94e61a4107e0bf48ae9f8; no voters:
I20260928 06:23:53.760591 8239 raft_consensus.cc:2781] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:53.760648 8239 raft_consensus.cc:686] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 1 LEADER]: Becoming Leader. State: Replica: 07eca35f332a45d9a407f793345ee8ab, State: Running, Role: LEADER
I20260928 06:23:53.760705 8239 consensus_queue.cc:237] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.761438 7686 catalog_manager.cc:5495] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 07eca35f332a45d9a407f793345ee8ab (127.31.214.65). New cstate: current_term: 1 leader_uuid: "07eca35f332a45d9a407f793345ee8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } health_report { overall_health: UNKNOWN } } }
I20260928 06:23:53.774829 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.772510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.31.214.65:41123)
0928 06:23:53.772557 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.772563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.772572 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:23:53.772595 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.772595 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.772605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.772655 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:53.772873 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.773359 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.774225 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.774426 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.774491 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.774604 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:53.774630 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:53.774673 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.774718 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.774735 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:23:53.775837 7983 raft_consensus.cc:1261] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Refusing update from remote peer 07eca35f332a45d9a407f793345ee8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.775964 8119 raft_consensus.cc:1261] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Refusing update from remote peer 07eca35f332a45d9a407f793345ee8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.776125 8204 consensus_queue.cc:1035] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.776242 8239 consensus_queue.cc:1035] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.779632 7686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:53.779743 7686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:53.780405 7686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2: Sending DeleteTablet for 3 replicas of tablet ede26f1559a745408fe8c98f4aa4b106
I20260928 06:23:53.780690 7823 tablet_service.cc:1514] Processing DeleteTablet for tablet ede26f1559a745408fe8c98f4aa4b106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:39420
I20260928 06:23:53.780794 8221 tablet_replica.cc:330] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: stopping tablet replica
I20260928 06:23:53.780774 7963 tablet_service.cc:1514] Processing DeleteTablet for tablet ede26f1559a745408fe8c98f4aa4b106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:35354
I20260928 06:23:53.780841 8221 raft_consensus.cc:2229] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:23:53.780864 8222 tablet_replica.cc:330] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: stopping tablet replica
I20260928 06:23:53.780901 8222 raft_consensus.cc:2229] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:53.780941 8222 raft_consensus.cc:2258] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.780938 8221 raft_consensus.cc:2258] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.781075 8103 tablet_service.cc:1514] Processing DeleteTablet for tablet ede26f1559a745408fe8c98f4aa4b106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:51996
I20260928 06:23:53.781199 8223 tablet_replica.cc:330] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: stopping tablet replica
I20260928 06:23:53.781262 8223 raft_consensus.cc:2229] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:53.781311 8223 raft_consensus.cc:2258] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.781311 7686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:23:53.781494 7686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:23:53.781785 8221 ts_tablet_manager.cc:1905] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.781805 8223 ts_tablet_manager.cc:1905] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.782228 8222 ts_tablet_manager.cc:1905] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.783560 8223 ts_tablet_manager.cc:1918] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:53.783615 8223 log.cc:1196] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/ede26f1559a745408fe8c98f4aa4b106
I20260928 06:23:53.783769 8222 ts_tablet_manager.cc:1918] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:53.783823 8222 log.cc:1196] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/ede26f1559a745408fe8c98f4aa4b106
I20260928 06:23:53.783849 8223 ts_tablet_manager.cc:1939] T ede26f1559a745408fe8c98f4aa4b106 P 905d452b01014e4aa80567ef7bec7e46: Deleting consensus metadata
I20260928 06:23:53.783996 8222 ts_tablet_manager.cc:1939] T ede26f1559a745408fe8c98f4aa4b106 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting consensus metadata
I20260928 06:23:53.784528 7671 catalog_manager.cc:4841] TS 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715): tablet ede26f1559a745408fe8c98f4aa4b106 (table test-table [id=01a688154af94665ac70ef3a06f3d5cd]) successfully deleted
I20260928 06:23:53.784749 7671 catalog_manager.cc:4841] TS 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415): tablet ede26f1559a745408fe8c98f4aa4b106 (table test-table [id=01a688154af94665ac70ef3a06f3d5cd]) successfully deleted
I20260928 06:23:53.785511 8221 ts_tablet_manager.cc:1918] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:23:53.785574 8221 log.cc:1196] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/ede26f1559a745408fe8c98f4aa4b106
I20260928 06:23:53.785794 8221 ts_tablet_manager.cc:1939] T ede26f1559a745408fe8c98f4aa4b106 P 07eca35f332a45d9a407f793345ee8ab: Deleting consensus metadata
I20260928 06:23:53.786305 7671 catalog_manager.cc:4841] TS 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123): tablet ede26f1559a745408fe8c98f4aa4b106 (table test-table [id=01a688154af94665ac70ef3a06f3d5cd]) successfully deleted
I20260928 06:23:53.786732 7963 tablet_service.cc:1467] Processing CreateTablet for tablet d8b4b46364804ef29368cc6c2da96446 (DEFAULT_TABLE table=test-table [id=22ced22f2c87438fa1410a6d42929a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.786780 8103 tablet_service.cc:1467] Processing CreateTablet for tablet d8b4b46364804ef29368cc6c2da96446 (DEFAULT_TABLE table=test-table [id=22ced22f2c87438fa1410a6d42929a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.786854 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8b4b46364804ef29368cc6c2da96446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.786837 7823 tablet_service.cc:1467] Processing CreateTablet for tablet d8b4b46364804ef29368cc6c2da96446 (DEFAULT_TABLE table=test-table [id=22ced22f2c87438fa1410a6d42929a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:23:53.786871 8103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8b4b46364804ef29368cc6c2da96446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.786942 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8b4b46364804ef29368cc6c2da96446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:53.788040 8198 tablet_bootstrap.cc:492] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: Bootstrap starting.
I20260928 06:23:53.788040 8201 tablet_bootstrap.cc:492] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: Bootstrap starting.
I20260928 06:23:53.788103 8197 tablet_bootstrap.cc:492] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab: Bootstrap starting.
I20260928 06:23:53.788394 8197 tablet_bootstrap.cc:654] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.788399 8198 tablet_bootstrap.cc:654] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.788463 8201 tablet_bootstrap.cc:654] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:53.789006 8198 tablet_bootstrap.cc:492] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: No bootstrap required, opened a new log
I20260928 06:23:53.789055 8198 ts_tablet_manager.cc:1397] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:53.789165 8201 tablet_bootstrap.cc:492] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: No bootstrap required, opened a new log
I20260928 06:23:53.789168 8198 raft_consensus.cc:348] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.789225 8201 ts_tablet_manager.cc:1397] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:53.789240 8198 raft_consensus.cc:374] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.789263 8198 raft_consensus.cc:729] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905d452b01014e4aa80567ef7bec7e46, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.789317 8198 consensus_queue.cc:260] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.789410 8198 ts_tablet_manager.cc:1428] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:23:53.789389 8201 raft_consensus.cc:348] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.789469 8201 raft_consensus.cc:374] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.789495 8201 raft_consensus.cc:729] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7441bf06a3f94e61a4107e0bf48ae9f8, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.789544 8201 consensus_queue.cc:260] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.789661 8201 ts_tablet_manager.cc:1428] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:23:53.790081 8197 tablet_bootstrap.cc:492] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab: No bootstrap required, opened a new log
I20260928 06:23:53.790132 8197 ts_tablet_manager.cc:1397] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 06:23:53.790227 8197 raft_consensus.cc:348] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.790304 8197 raft_consensus.cc:374] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:53.790325 8197 raft_consensus.cc:729] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07eca35f332a45d9a407f793345ee8ab, State: Initialized, Role: FOLLOWER
I20260928 06:23:53.790369 8197 consensus_queue.cc:260] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.790459 8197 ts_tablet_manager.cc:1428] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:53.854081 8203 raft_consensus.cc:482] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:23:53.854206 8203 raft_consensus.cc:504] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.854393 8203 leader_election.cc:288] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123), 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415)
I20260928 06:23:53.854660 7843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8b4b46364804ef29368cc6c2da96446" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07eca35f332a45d9a407f793345ee8ab" is_pre_election: true
I20260928 06:23:53.854696 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8b4b46364804ef29368cc6c2da96446" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" is_pre_election: true
I20260928 06:23:53.854772 7843 raft_consensus.cc:2454] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 0.
I20260928 06:23:53.854821 7983 raft_consensus.cc:2454] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 0.
I20260928 06:23:53.855000 8055 leader_election.cc:302] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7441bf06a3f94e61a4107e0bf48ae9f8, 905d452b01014e4aa80567ef7bec7e46; no voters:
I20260928 06:23:53.855134 8203 raft_consensus.cc:2781] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:23:53.855197 8203 raft_consensus.cc:482] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:23:53.855221 8203 raft_consensus.cc:3037] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.856144 8203 raft_consensus.cc:504] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.856284 8203 leader_election.cc:288] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 election: Requested vote from peers 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123), 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415)
I20260928 06:23:53.856513 7843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8b4b46364804ef29368cc6c2da96446" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07eca35f332a45d9a407f793345ee8ab"
I20260928 06:23:53.856609 7843 raft_consensus.cc:3037] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.856570 7983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8b4b46364804ef29368cc6c2da96446" candidate_uuid: "905d452b01014e4aa80567ef7bec7e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8"
I20260928 06:23:53.856650 7983 raft_consensus.cc:3037] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:53.857313 7843 raft_consensus.cc:2454] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 1.
I20260928 06:23:53.857429 7983 raft_consensus.cc:2454] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905d452b01014e4aa80567ef7bec7e46 in term 1.
I20260928 06:23:53.857514 8055 leader_election.cc:302] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07eca35f332a45d9a407f793345ee8ab, 905d452b01014e4aa80567ef7bec7e46; no voters:
I20260928 06:23:53.857618 8203 raft_consensus.cc:2781] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:53.857671 8203 raft_consensus.cc:686] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 1 LEADER]: Becoming Leader. State: Replica: 905d452b01014e4aa80567ef7bec7e46, State: Running, Role: LEADER
I20260928 06:23:53.857725 8203 consensus_queue.cc:237] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } }
I20260928 06:23:53.858386 7685 catalog_manager.cc:5495] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67). New cstate: current_term: 1 leader_uuid: "905d452b01014e4aa80567ef7bec7e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "905d452b01014e4aa80567ef7bec7e46" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 36715 } health_report { overall_health: HEALTHY } } }
I20260928 06:23:53.870628 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.868227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.31.214.67:36715)
0928 06:23:53.868269 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:53.868271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:53.868280 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:23:53.868334 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:53.868335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:53.868352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:53.868418 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:23:53.868509 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.869200 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.869917 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:53.870225 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:53.870295 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:53.870405 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:23:53.870436 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:23:53.870482 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:53.870534 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:23:53.870554 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:23:53.873099 7843 raft_consensus.cc:1261] T d8b4b46364804ef29368cc6c2da96446 P 07eca35f332a45d9a407f793345ee8ab [term 1 FOLLOWER]: Refusing update from remote peer 905d452b01014e4aa80567ef7bec7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.873101 7983 raft_consensus.cc:1261] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Refusing update from remote peer 905d452b01014e4aa80567ef7bec7e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:23:53.873361 8218 consensus_queue.cc:1035] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7441bf06a3f94e61a4107e0bf48ae9f8" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.873492 8218 consensus_queue.cc:1035] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07eca35f332a45d9a407f793345ee8ab" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:23:53.884531 7686 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:53.884660 7686 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:23:53.885315 7686 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97a2f9fafc2643cfbc3094e29f2b24e2: Sending DeleteTablet for 3 replicas of tablet d8b4b46364804ef29368cc6c2da96446
I20260928 06:23:53.885636 7963 tablet_service.cc:1514] Processing DeleteTablet for tablet d8b4b46364804ef29368cc6c2da96446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:35354
I20260928 06:23:53.885757 8222 tablet_replica.cc:330] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: stopping tablet replica
I20260928 06:23:53.885839 8222 raft_consensus.cc:2229] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:23:53.885874 8222 raft_consensus.cc:2258] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.885846 7823 tablet_service.cc:1514] Processing DeleteTablet for tablet d8b4b46364804ef29368cc6c2da96446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:39420
I20260928 06:23:53.886202 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 7758
I20260928 06:23:53.886215 8222 ts_tablet_manager.cc:1905] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.887822 8222 ts_tablet_manager.cc:1918] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:53.887881 8222 log.cc:1196] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/d8b4b46364804ef29368cc6c2da96446
I20260928 06:23:53.888165 8222 ts_tablet_manager.cc:1939] T d8b4b46364804ef29368cc6c2da96446 P 7441bf06a3f94e61a4107e0bf48ae9f8: Deleting consensus metadata
I20260928 06:23:53.888535 7671 catalog_manager.cc:4841] TS 7441bf06a3f94e61a4107e0bf48ae9f8 (127.31.214.66:33415): tablet d8b4b46364804ef29368cc6c2da96446 (table test-table [id=22ced22f2c87438fa1410a6d42929a2f]) successfully deleted
I20260928 06:23:53.889938 8103 tablet_service.cc:1514] Processing DeleteTablet for tablet d8b4b46364804ef29368cc6c2da96446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:23:53 UTC) from {username='oryx', principal='oryx/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:51996
I20260928 06:23:53.890089 8223 tablet_replica.cc:330] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: stopping tablet replica
I20260928 06:23:53.890168 8223 raft_consensus.cc:2229] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:23:53.890249 8223 raft_consensus.cc:2258] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:23:53.890535 8223 ts_tablet_manager.cc:1905] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:23:53.892014 8223 ts_tablet_manager.cc:1918] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:23:53.892107 8223 log.cc:1196] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/d8b4b46364804ef29368cc6c2da96446
I20260928 06:23:53.892340 8223 ts_tablet_manager.cc:1939] T d8b4b46364804ef29368cc6c2da96446 P 905d452b01014e4aa80567ef7bec7e46: Deleting consensus metadata
I20260928 06:23:53.892724 7671 catalog_manager.cc:4841] TS 905d452b01014e4aa80567ef7bec7e46 (127.31.214.67:36715): tablet d8b4b46364804ef29368cc6c2da96446 (table test-table [id=22ced22f2c87438fa1410a6d42929a2f]) successfully deleted
W20260928 06:23:53.895092 7671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:41123 (error 108)
I20260928 06:23:53.895334 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 7898
I20260928 06:23:53.896363 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:53.896183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:41123 (local address 127.0.0.1:39442)
0928 06:23:53.896281 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:53.896325 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:41123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":46,"threads_started":1}
W20260928 06:23:53.896525 7671 catalog_manager.cc:4584] TS 07eca35f332a45d9a407f793345ee8ab (127.31.214.65:41123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8b4b46364804ef29368cc6c2da96446: Network error: Client connection negotiation failed: client connection to 127.31.214.65:41123: connect: Connection refused (error 111)
I20260928 06:23:53.901927 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 8038
I20260928 06:23:53.908469 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 7654
2026-09-28T06:23:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2520 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[8283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[8283](info): set up 2 sockets
Sep 28 06:23:53 dist-test-slave-q03f krb5kdc[8283](info): commencing operation
krb5kdc: starting...
W20260928 06:23:55.030635 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:55 dist-test-slave-q03f krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:23:55.162467 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:42121
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:42121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:55.251084 8296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:55.251348 8296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:55.251377 8296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:55.252638 8296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:55.252689 8296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:55.252707 8296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:55.252768 8296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:55.252789 8296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:55.254423 8296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:42121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:42121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:55.254717 8296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:55.254979 8296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:55.260424 8304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:55.260440 8302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:55.260447 8301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:55.260584 8296 server_base.cc:1034] running on GCE node
I20260928 06:23:55.261078 8296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:55.261453 8296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:55.262643 8296 hybrid_clock.cc:648] HybridClock initialized: now 1790576635262618 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T06:23:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1360 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:23:55 dist-test-slave-q03f krb5kdc[8309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:23:55 dist-test-slave-q03f krb5kdc[8309](info): set up 2 sockets
Sep 28 06:23:55 dist-test-slave-q03f krb5kdc[8309](info): commencing operation
krb5kdc: starting...
W20260928 06:23:56.322561 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:23:56 dist-test-slave-q03f krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:23:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:23:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:23:56.454952 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:43267
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:43267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:56.543257 8322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:56.543490 8322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:56.543516 8322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:56.544665 8322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:56.544710 8322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:56.544725 8322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:56.544740 8322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:56.544752 8322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:56.546406 8322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:43267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:43267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:56.546629 8322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:56.546823 8322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:56.552175 8330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:56.552175 8328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:56.552265 8322 server_base.cc:1034] running on GCE node
W20260928 06:23:56.552202 8327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:56.552704 8322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:56.553062 8322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:56.554235 8322 hybrid_clock.cc:648] HybridClock initialized: now 1790576636554213 us; error 42 us; skew 500 ppm
Sep 28 06:23:56 dist-test-slave-q03f krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576636, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:56.564664 8322 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:23:56.565264 8322 webserver.cc:460] Webserver started at http://127.31.214.126:38819/ using document root <none> and password file <none>
I20260928 06:23:56.565539 8322 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:56.565614 8322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:56.565773 8322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:56.566999 8322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "58baf12d4fb54801b8e0c2653b5c3e38"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "618b9b6ef8de98a4ab1300a1324a4c12"
server_key_iv: "c4bd4b86a98055eb548bbe5cc6ddd314"
server_key_version: "encryptionkey@0"
I20260928 06:23:56.567389 8322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "58baf12d4fb54801b8e0c2653b5c3e38"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "618b9b6ef8de98a4ab1300a1324a4c12"
server_key_iv: "c4bd4b86a98055eb548bbe5cc6ddd314"
server_key_version: "encryptionkey@0"
I20260928 06:23:56.568826 8322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:23:56.569628 8337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:56.570164 8322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:56.570286 8322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "58baf12d4fb54801b8e0c2653b5c3e38"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "618b9b6ef8de98a4ab1300a1324a4c12"
server_key_iv: "c4bd4b86a98055eb548bbe5cc6ddd314"
server_key_version: "encryptionkey@0"
I20260928 06:23:56.570366 8322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:56.595872 8322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:56.598591 8322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:56.598783 8322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:56.603210 8322 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:43267
I20260928 06:23:56.603251 8389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:43267 every 8 connection(s)
I20260928 06:23:56.603665 8322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:56.604590 8390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:23:56.607198 8390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: Bootstrap starting.
I20260928 06:23:56.607841 8390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: Neither blocks nor log segments found. Creating new log.
I20260928 06:23:56.608165 8390 log.cc:824] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:56.609104 8390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: No bootstrap required, opened a new log
I20260928 06:23:56.610986 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 8322
I20260928 06:23:56.611127 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:23:56.611347 32601 external_mini_cluster.cc:1428] Setting key 4ba1b144d2f4b28e81392a8b18606638
I20260928 06:23:56.611410 8390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } }
I20260928 06:23:56.611645 8390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:23:56.611701 8390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58baf12d4fb54801b8e0c2653b5c3e38, State: Initialized, Role: FOLLOWER
I20260928 06:23:56.611917 8390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } }
I20260928 06:23:56.612030 8390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:56.612098 8390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:56.612166 8390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:23:56.613566 8390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } }
I20260928 06:23:56.613700 8390 leader_election.cc:302] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58baf12d4fb54801b8e0c2653b5c3e38; no voters:
I20260928 06:23:56.613884 8390 leader_election.cc:288] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:23:56.613922 8395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:23:56.614066 8395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 1 LEADER]: Becoming Leader. State: Replica: 58baf12d4fb54801b8e0c2653b5c3e38, State: Running, Role: LEADER
I20260928 06:23:56.614156 8395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } }
I20260928 06:23:56.614207 8390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:56.614537 8395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } } }
I20260928 06:23:56.614619 8395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:56.614744 8395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58baf12d4fb54801b8e0c2653b5c3e38. Latest consensus state: current_term: 1 leader_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } } }
I20260928 06:23:56.614813 8395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:56.614981 8399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:56.615567 8399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:56.618043 8399 catalog_manager.cc:1347] Generated new cluster ID: 9259ce302e6c4a458c6db2454c792aeb
I20260928 06:23:56.618098 8399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:23:56 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:56.624182 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:56.612099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.31.214.126:43267)
0928 06:23:56.612399 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:56.612409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:56.612440 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:23:56.612552 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:56.612559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:56.612615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:56.612744 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:56.613751 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.615244 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.616318 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.616722 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.616798 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:56.619946 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:56.619951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:56.619952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:56.619965 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:56.622245 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:56.622666 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:56.622670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:56.622670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:56.622808 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:56.623036 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:56.623037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:56.623038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:56.623403 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:56.623595 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:56.623681 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 06:23:56.623892 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
I20260928 06:23:56.627293 8399 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:23:56.627842 8399 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:56.634294 8399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: Generated new TSK 0
I20260928 06:23:56.634500 8399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:23:56.666064 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43267
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:56.754444 8416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:56.754671 8416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:56.754698 8416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:56.755901 8416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:56.755947 8416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:56.755975 8416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:56.757447 8416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:56.757715 8416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:56.758040 8416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:56.763212 8424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:56.763217 8421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:56.763242 8422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:56.763583 8416 server_base.cc:1034] running on GCE node
I20260928 06:23:56.763818 8416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:56.764079 8416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:56.765256 8416 hybrid_clock.cc:648] HybridClock initialized: now 1790576636765232 us; error 34 us; skew 500 ppm
Sep 28 06:23:56 dist-test-slave-q03f krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576636, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:56.767865 8416 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:23:56.768327 8416 webserver.cc:460] Webserver started at http://127.31.214.65:42245/ using document root <none> and password file <none>
I20260928 06:23:56.768539 8416 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:56.768580 8416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:56.768671 8416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:56.769677 8416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "2bdcf53bb4ec4287a4cee3b9f6d3c08b"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "c895ca67408910a9688c685e4b72def2"
server_key_iv: "51a51943aa4acecd5cfa92d9765280a9"
server_key_version: "encryptionkey@0"
I20260928 06:23:56.770056 8416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "2bdcf53bb4ec4287a4cee3b9f6d3c08b"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "c895ca67408910a9688c685e4b72def2"
server_key_iv: "51a51943aa4acecd5cfa92d9765280a9"
server_key_version: "encryptionkey@0"
I20260928 06:23:56.771400 8416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:56.772136 8431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:56.772379 8416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:56.772459 8416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "2bdcf53bb4ec4287a4cee3b9f6d3c08b"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "c895ca67408910a9688c685e4b72def2"
server_key_iv: "51a51943aa4acecd5cfa92d9765280a9"
server_key_version: "encryptionkey@0"
I20260928 06:23:56.772513 8416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:56.786386 8416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:56.788944 8416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:56.789068 8416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:56.789264 8416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:56.789611 8416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:56.789651 8416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:56.789675 8416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:56.789687 8416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:56.796132 8416 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:45129
I20260928 06:23:56.796300 8544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:45129 every 8 connection(s)
I20260928 06:23:56.796571 8416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:23:56.801776 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 8416
I20260928 06:23:56.802012 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:23:56.802241 32601 external_mini_cluster.cc:1428] Setting key e2bfe04d6aa33a8342a642746158f4d8
Sep 28 06:23:56 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576636, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:56.807266 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:56.797435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:60897 (local address 127.31.214.126:43267)
0928 06:23:56.797557 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:56.797560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:56.797761 (+ 201us) server_negotiation.cc:407] Connection header received
0928 06:23:56.798402 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:56.798404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:56.798422 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:56.798509 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:56.799589 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.800169 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.801442 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.801677 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.801785 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:56.804281 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:56.804283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:56.804283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:56.804293 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:56.805741 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:56.806241 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:56.806243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:56.806243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:56.806306 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:56.806594 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:56.806595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:56.806596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:56.806712 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:56.806792 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:56.807087 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:23:56.807175 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 06:23:56.807457 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:56.797295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:60897)
0928 06:23:56.797596 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:56.797647 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:56.798263 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:56.798600 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:56.798603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:56.799244 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:56.799455 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:56.799460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.800303 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.800304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:56.801320 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:56.801323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.801797 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.801799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:56.802007 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:56.802676 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:56.802686 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:56.804168 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:56.805843 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:56.805851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:56.805853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:56.806147 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:56.806393 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:56.806393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:56.806394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:56.806517 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:56.806782 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:56.806789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:56.806944 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 06:23:56.807053 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:23:56.807214 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":116,"threads_started":1}
I20260928 06:23:56.807992 8545 heartbeater.cc:347] Connected to a master server at 127.31.214.126:43267
I20260928 06:23:56.808130 8545 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:56.808347 8545 heartbeater.cc:511] Master 127.31.214.126:43267 requested a full tablet report, sending...
I20260928 06:23:56.808926 8354 ts_manager.cc:194] Registered new tserver with Master: 2bdcf53bb4ec4287a4cee3b9f6d3c08b (127.31.214.65:45129)
I20260928 06:23:56.809572 8354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:60897
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:23:56.833516 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43267
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:56.920920 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:56.921128 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:56.921152 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:56.922346 8550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:56.922394 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:56.922420 8550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:56.924187 8550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:56.924399 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:56.924593 8550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:56.930294 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:56.930290 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:56.930285 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:56.930586 8550 server_base.cc:1034] running on GCE node
I20260928 06:23:56.930801 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:56.931056 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:56.932243 8550 hybrid_clock.cc:648] HybridClock initialized: now 1790576636932235 us; error 37 us; skew 500 ppm
Sep 28 06:23:56 dist-test-slave-q03f krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576636, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:56.935204 8550 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:23:56.935676 8550 webserver.cc:460] Webserver started at http://127.31.214.66:43547/ using document root <none> and password file <none>
I20260928 06:23:56.935887 8550 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:56.935933 8550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:56.936034 8550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:56.937163 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "a02b8ecd40834e3f911f47e9f0b9e269"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "b1072843482e7b3f1f3b2be1068e386d"
server_key_iv: "445042c3b04c9babad8b4d1bcaa8b4aa"
server_key_version: "encryptionkey@0"
I20260928 06:23:56.937523 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "a02b8ecd40834e3f911f47e9f0b9e269"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "b1072843482e7b3f1f3b2be1068e386d"
server_key_iv: "445042c3b04c9babad8b4d1bcaa8b4aa"
server_key_version: "encryptionkey@0"
I20260928 06:23:56.939121 8550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:23:56.940048 8565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:56.940258 8550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:23:56.940344 8550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "a02b8ecd40834e3f911f47e9f0b9e269"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "b1072843482e7b3f1f3b2be1068e386d"
server_key_iv: "445042c3b04c9babad8b4d1bcaa8b4aa"
server_key_version: "encryptionkey@0"
I20260928 06:23:56.940412 8550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:56.953771 8550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:56.955981 8550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:56.956105 8550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:56.956424 8550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:56.956758 8550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:56.956801 8550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:56.956830 8550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:56.956846 8550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:56.963184 8550 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:45273
I20260928 06:23:56.963243 8678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:45273 every 8 connection(s)
I20260928 06:23:56.963624 8550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:23:56.968978 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 8550
I20260928 06:23:56.969162 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:23:56.969419 32601 external_mini_cluster.cc:1428] Setting key 9b2d026962045115351101cb2ca41247
Sep 28 06:23:56 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576636, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:56.974366 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:56.964489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:40835 (local address 127.31.214.126:43267)
0928 06:23:56.964582 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:56.964585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:56.964791 (+ 206us) server_negotiation.cc:407] Connection header received
0928 06:23:56.965271 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:56.965272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:56.965282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:56.965345 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:56.966403 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.966995 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.968087 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.968278 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.968332 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:56.970943 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:56.970945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:56.970945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:56.970954 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:56.972635 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:56.973166 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:56.973167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:56.973167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:56.973253 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:56.973550 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:56.973551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:56.973551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:56.973663 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:56.973749 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:56.974137 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 06:23:56.974246 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:23:56.974728 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:56.964401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:40835)
0928 06:23:56.964654 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:56.964698 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:56.965186 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:56.965403 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:56.965408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:56.966091 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:56.966305 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:56.966311 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.967107 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.967108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:56.968001 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:56.968004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:56.968367 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:56.968368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:56.968478 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:56.969171 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:56.969182 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:56.970838 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:56.972717 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:56.972725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:56.972727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:56.973084 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:56.973327 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:56.973328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:56.973329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:56.973482 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:56.973768 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:56.973774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:56.973990 (+ 216us) client_negotiation.cc:769] Sending connection context
0928 06:23:56.974116 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 06:23:56.974388 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":123,"threads_started":1}
I20260928 06:23:56.975273 8679 heartbeater.cc:347] Connected to a master server at 127.31.214.126:43267
I20260928 06:23:56.975389 8679 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:56.975636 8679 heartbeater.cc:511] Master 127.31.214.126:43267 requested a full tablet report, sending...
I20260928 06:23:56.976078 8354 ts_manager.cc:194] Registered new tserver with Master: a02b8ecd40834e3f911f47e9f0b9e269 (127.31.214.66:45273)
I20260928 06:23:56.976475 8354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:40835
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:23:57.002332 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43267
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:57.086870 8684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:57.087107 8684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:57.087132 8684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:57.088279 8684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:57.088320 8684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:57.088346 8684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:57.089752 8684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:57.090019 8684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:57.090245 8684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:57.095710 8690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.095832 8689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.095710 8692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:57.095896 8684 server_base.cc:1034] running on GCE node
I20260928 06:23:57.096114 8684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:57.096377 8684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:57.097529 8684 hybrid_clock.cc:648] HybridClock initialized: now 1790576637097505 us; error 45 us; skew 500 ppm
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576637, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:57.100423 8684 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:23:57.100975 8684 webserver.cc:460] Webserver started at http://127.31.214.67:42871/ using document root <none> and password file <none>
I20260928 06:23:57.101238 8684 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:57.101310 8684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:57.101475 8684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:23:57.102660 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "d88647e989284ab2a1a479c38ed643a9"
format_stamp: "Formatted at 2026-09-28 06:23:57 on dist-test-slave-q03f"
server_key: "610bbb8174d21104d7b27b239073ddc4"
server_key_iv: "31d04dd40aacff69e71937406edf1b65"
server_key_version: "encryptionkey@0"
I20260928 06:23:57.103040 8684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "d88647e989284ab2a1a479c38ed643a9"
format_stamp: "Formatted at 2026-09-28 06:23:57 on dist-test-slave-q03f"
server_key: "610bbb8174d21104d7b27b239073ddc4"
server_key_iv: "31d04dd40aacff69e71937406edf1b65"
server_key_version: "encryptionkey@0"
I20260928 06:23:57.104467 8684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:23:57.105327 8699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.105522 8684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:57.105631 8684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "d88647e989284ab2a1a479c38ed643a9"
format_stamp: "Formatted at 2026-09-28 06:23:57 on dist-test-slave-q03f"
server_key: "610bbb8174d21104d7b27b239073ddc4"
server_key_iv: "31d04dd40aacff69e71937406edf1b65"
server_key_version: "encryptionkey@0"
I20260928 06:23:57.105716 8684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:57.134792 8684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:57.137269 8684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:57.137444 8684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:57.137705 8684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:57.138198 8684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:57.138266 8684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.138304 8684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:57.138324 8684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.144433 8684 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:38757
I20260928 06:23:57.144548 8812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:38757 every 8 connection(s)
I20260928 06:23:57.144855 8684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:57.147843 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 8684
I20260928 06:23:57.147986 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:23:57.148168 32601 external_mini_cluster.cc:1428] Setting key 4b2191ab5ef83b2efd985109ba59f7ee
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576637, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:23:57.155144 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.145705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:38069 (local address 127.31.214.126:43267)
0928 06:23:57.145771 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.145773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.146030 (+ 257us) server_negotiation.cc:407] Connection header received
0928 06:23:57.146581 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.146583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.146594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.146658 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.147723 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.148416 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.149687 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.149874 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.149930 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.152168 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:57.152170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:57.152170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:57.152177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:57.153692 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:57.154184 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:57.154186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:57.154186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:57.154245 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:57.154519 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:57.154519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:57.154520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:57.154626 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:57.154672 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:57.155000 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 06:23:57.155078 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:23:57.155196 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.145679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:38069)
0928 06:23:57.145854 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.145900 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.146455 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.146731 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.146740 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.147379 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.147585 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.147590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.148538 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.148540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.149607 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.149610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.150037 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.150038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.150164 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.150630 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.150638 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.152079 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:23:57.153773 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:57.153779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:57.153781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:57.154091 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:57.154342 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:23:57.154343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:23:57.154343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:23:57.154446 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:23:57.154678 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:23:57.154682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:23:57.154803 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 06:23:57.154883 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:23:57.155020 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260928 06:23:57.155779 8813 heartbeater.cc:347] Connected to a master server at 127.31.214.126:43267
I20260928 06:23:57.155900 8813 heartbeater.cc:464] Registering TS with master...
I20260928 06:23:57.156104 8813 heartbeater.cc:511] Master 127.31.214.126:43267 requested a full tablet report, sending...
I20260928 06:23:57.156529 8354 ts_manager.cc:194] Registered new tserver with Master: d88647e989284ab2a1a479c38ed643a9 (127.31.214.67:38757)
I20260928 06:23:57.157003 8354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:38069
I20260928 06:23:57.160051 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab.
Entry for principal oryx/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab.
WARNING: no policy specified for oryx/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab.
Entry for principal oryx/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab.
WARNING: no policy specified for oryx/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab.
Entry for principal oryx/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab.
I20260928 06:23:57.245889 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 8416
I20260928 06:23:57.252592 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 8550
I20260928 06:23:57.257318 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 8684
I20260928 06:23:57.262030 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 8322
I20260928 06:23:57.267596 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:43267
--webserver_interface=127.31.214.126
--webserver_port=38819
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:43267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:23:57.355258 8822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:57.355484 8822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:57.355507 8822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:57.356739 8822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:23:57.356789 8822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:57.356803 8822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:57.356817 8822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:23:57.356828 8822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:23:57.358430 8822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:43267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:43267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=38819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:57.358734 8822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:57.358986 8822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:57.364286 8830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.364321 8827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.364337 8828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:57.364480 8822 server_base.cc:1034] running on GCE node
I20260928 06:23:57.364956 8822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:57.365291 8822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:57.366479 8822 hybrid_clock.cc:648] HybridClock initialized: now 1790576637366461 us; error 52 us; skew 500 ppm
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576637, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:57.369184 8822 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:23:57.369808 8822 webserver.cc:460] Webserver started at http://127.31.214.126:38819/ using document root <none> and password file <none>
I20260928 06:23:57.370100 8822 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:57.370188 8822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:57.371632 8822 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:57.372356 8837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.372534 8822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:57.372635 8822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "58baf12d4fb54801b8e0c2653b5c3e38"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "618b9b6ef8de98a4ab1300a1324a4c12"
server_key_iv: "c4bd4b86a98055eb548bbe5cc6ddd314"
server_key_version: "encryptionkey@0"
I20260928 06:23:57.373013 8822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:57.386762 8822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:57.388857 8822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:57.389037 8822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:57.393397 8822 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:43267
I20260928 06:23:57.393442 8889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:43267 every 8 connection(s)
I20260928 06:23:57.393860 8822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:23:57.395465 8890 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:23:57.396150 8890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: Bootstrap starting.
I20260928 06:23:57.403460 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 8822
I20260928 06:23:57.403949 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:45129
--local_ip_for_outbound_sockets=127.31.214.65
--tserver_master_addrs=127.31.214.126:43267
--webserver_port=42245
--webserver_interface=127.31.214.65
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.214.65
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:23:57.404688 8890 log.cc:824] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: Log is configured to *not* fsync() on all Append() calls
I20260928 06:23:57.407235 8890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:23:57.407558 8890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: Bootstrap complete.
I20260928 06:23:57.411701 8890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } }
I20260928 06:23:57.412122 8890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58baf12d4fb54801b8e0c2653b5c3e38, State: Initialized, Role: FOLLOWER
I20260928 06:23:57.412312 8890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } }
I20260928 06:23:57.412403 8890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:23:57.412452 8890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:23:57.412514 8890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:23:57.414193 8890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } }
I20260928 06:23:57.414323 8890 leader_election.cc:302] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58baf12d4fb54801b8e0c2653b5c3e38; no voters:
I20260928 06:23:57.414525 8890 leader_election.cc:288] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:23:57.414606 8894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:23:57.414835 8890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:23:57.414816 8894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [term 2 LEADER]: Becoming Leader. State: Replica: 58baf12d4fb54801b8e0c2653b5c3e38, State: Running, Role: LEADER
I20260928 06:23:57.415107 8894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } }
I20260928 06:23:57.415522 8895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } } }
I20260928 06:23:57.415599 8895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:57.415894 8896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58baf12d4fb54801b8e0c2653b5c3e38. Latest consensus state: current_term: 2 leader_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58baf12d4fb54801b8e0c2653b5c3e38" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43267 } } }
I20260928 06:23:57.416028 8903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:23:57.416066 8896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38 [sys.catalog]: This master's current role is: LEADER
I20260928 06:23:57.416684 8903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:23:57.417039 8903 catalog_manager.cc:1259] Loaded cluster ID: 9259ce302e6c4a458c6db2454c792aeb
I20260928 06:23:57.417085 8903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:23:57.418311 8903 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:23:57.418507 8903 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 58baf12d4fb54801b8e0c2653b5c3e38: Loaded TSK: 0
I20260928 06:23:57.418727 8903 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:23:57.492125 8893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:57.492379 8893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:57.492421 8893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:57.493712 8893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:57.493790 8893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:57.493839 8893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:23:57.495466 8893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.65.keytab
--principal=oryx/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:45129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=42245
--tserver_master_addrs=127.31.214.126:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:57.495771 8893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:57.496042 8893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:57.501535 8919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.501523 8917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.501523 8916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:57.502017 8893 server_base.cc:1034] running on GCE node
I20260928 06:23:57.502247 8893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:57.502538 8893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:57.503715 8893 hybrid_clock.cc:648] HybridClock initialized: now 1790576637503695 us; error 37 us; skew 500 ppm
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576637, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:57.506598 8893 init.cc:377] Logged in from keytab as oryx/127.31.214.65@KRBTEST.COM (short username oryx)
I20260928 06:23:57.507130 8893 webserver.cc:460] Webserver started at http://127.31.214.65:42245/ using document root <none> and password file <none>
I20260928 06:23:57.507400 8893 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:57.507493 8893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:57.509048 8893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:57.509776 8926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.510040 8893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:57.510151 8893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "2bdcf53bb4ec4287a4cee3b9f6d3c08b"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "c895ca67408910a9688c685e4b72def2"
server_key_iv: "51a51943aa4acecd5cfa92d9765280a9"
server_key_version: "encryptionkey@0"
I20260928 06:23:57.510538 8893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:57.540124 8893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:57.542627 8893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:57.542819 8893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:57.543095 8893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:57.543429 8893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:57.543490 8893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.543532 8893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:57.543553 8893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.549666 8893 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:45129
I20260928 06:23:57.549710 9039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:45129 every 8 connection(s)
I20260928 06:23:57.550187 8893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:23:57.550467 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 8893
I20260928 06:23:57.550791 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:45273
--local_ip_for_outbound_sockets=127.31.214.66
--tserver_master_addrs=127.31.214.126:43267
--webserver_port=43547
--webserver_interface=127.31.214.66
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.214.66
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:57.560961 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.551383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:59185)
0928 06:23:57.551615 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.551666 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.552456 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.552998 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.553002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.553653 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.553930 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.553936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.555748 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.555749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.556585 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.556587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.557519 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.557520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.557625 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.558271 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.558280 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.560802 (+ 2522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
W20260928 06:23:57.561218 9040 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:43267 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.214.126:43267: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
W20260928 06:23:57.561215 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:57.551383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:59185 (local address 127.31.214.126:43267)
0928 06:23:57.551608 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.551618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.551778 (+ 160us) server_negotiation.cc:407] Connection header received
0928 06:23:57.552677 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.552685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.552756 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.552915 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.554145 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.555622 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.556681 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.557408 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.557536 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.561001 (+ 3465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:59185: BlockingRecv error: recv got EOF from 127.31.214.65:59185 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:57.566450 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.561497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:54093)
0928 06:23:57.561557 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.561570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.561612 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.561790 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.561791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.561925 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.562021 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.562023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.563006 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.563007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.563502 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.563503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.563993 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.563994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.564044 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.564504 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.564509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.566379 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:23:57.566557 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:57.561590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:54093 (local address 127.31.214.126:43267)
0928 06:23:57.561650 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.561653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.561662 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:23:57.561688 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.561689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.561702 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.561774 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.562105 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.562902 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.563587 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.563902 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.563982 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.566474 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:54093: BlockingRecv error: recv got EOF from 127.31.214.65:54093 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:57.571350 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.566803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:49355)
0928 06:23:57.566857 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.566870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.566911 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.567060 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.567061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.567165 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.567191 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.567192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.568056 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.568057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.568545 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.568547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.568998 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.568999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.569037 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.569474 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.569478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.571281 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:23:57.571403 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:57.566886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:49355 (local address 127.31.214.126:43267)
0928 06:23:57.566948 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.566950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.566957 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:23:57.566978 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.566978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.566988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.567049 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.567255 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.567983 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.568620 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.568913 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.568983 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.571361 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:49355: BlockingRecv error: recv got EOF from 127.31.214.65:49355 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 06:23:57.571638 9040 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:23:57.640640 9044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:57.640863 9044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:57.640887 9044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:57.642207 9044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:57.642256 9044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:57.642283 9044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:23:57.643828 9044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.66.keytab
--principal=oryx/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:45273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=43547
--tserver_master_addrs=127.31.214.126:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:57.644114 9044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:57.644357 9044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:57.649710 9052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.649701 9050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.649683 9049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:57.650185 9044 server_base.cc:1034] running on GCE node
I20260928 06:23:57.650455 9044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:57.650705 9044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:57.651917 9044 hybrid_clock.cc:648] HybridClock initialized: now 1790576637651904 us; error 33 us; skew 500 ppm
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576637, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:57.654721 9044 init.cc:377] Logged in from keytab as oryx/127.31.214.66@KRBTEST.COM (short username oryx)
I20260928 06:23:57.655180 9044 webserver.cc:460] Webserver started at http://127.31.214.66:43547/ using document root <none> and password file <none>
I20260928 06:23:57.655428 9044 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:57.655494 9044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:57.657017 9044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:57.657786 9059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.658041 9044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:23:57.658146 9044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "a02b8ecd40834e3f911f47e9f0b9e269"
format_stamp: "Formatted at 2026-09-28 06:23:56 on dist-test-slave-q03f"
server_key: "b1072843482e7b3f1f3b2be1068e386d"
server_key_iv: "445042c3b04c9babad8b4d1bcaa8b4aa"
server_key_version: "encryptionkey@0"
I20260928 06:23:57.658543 9044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:57.667662 9044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:57.669682 9044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:57.669840 9044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:57.670120 9044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:57.670437 9044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:57.670472 9044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.670495 9044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:57.670535 9044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.676645 9044 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:45273
I20260928 06:23:57.676689 9172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:45273 every 8 connection(s)
I20260928 06:23:57.677079 9044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:23:57.677592 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 9044
I20260928 06:23:57.677929 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:38757
--local_ip_for_outbound_sockets=127.31.214.67
--tserver_master_addrs=127.31.214.126:43267
--webserver_port=42871
--webserver_interface=127.31.214.67
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.214.67
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:57.686123 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.677844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:40619)
0928 06:23:57.678101 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.678150 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.678931 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.679196 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.679200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.679959 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.680164 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.680169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.681185 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.681187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.682081 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.682084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.682706 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.682707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.682823 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.683447 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.683455 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.685881 (+ 2426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
W20260928 06:23:57.686132 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:57.677836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:40619 (local address 127.31.214.126:43267)
0928 06:23:57.677888 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.677890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.678257 (+ 367us) server_negotiation.cc:407] Connection header received
0928 06:23:57.679051 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.679052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.679066 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.679121 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.680278 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.681082 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.682178 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.682607 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.682692 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.686044 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:40619: BlockingRecv error: recv got EOF from 127.31.214.66:40619 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:23:57.686493 9173 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:43267 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.214.126:43267: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:57.692401 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.687835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:59633)
0928 06:23:57.687885 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.687896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.687943 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.688113 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.688114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.688248 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.688296 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.688298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.688946 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.688947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.689457 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.689459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.689862 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.689862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.689936 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.690478 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.690482 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.692326 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:23:57.692529 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:57.687917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:59633 (local address 127.31.214.126:43267)
0928 06:23:57.687997 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.687999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.688007 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:23:57.688033 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.688034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.688044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.688094 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.688382 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.688861 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.689563 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.689772 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.689861 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.692415 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:59633: BlockingRecv error: recv got EOF from 127.31.214.66:59633 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:57.697409 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.692652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:54281)
0928 06:23:57.692829 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.692842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.692896 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.693033 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.693034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.693166 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.693219 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.693220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.694144 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.694145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.694717 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.694719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.695208 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.695209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.695254 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.695737 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.695741 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.697348 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
W20260928 06:23:57.697525 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:57.692846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:54281 (local address 127.31.214.126:43267)
0928 06:23:57.692918 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.692920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.692930 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:23:57.692951 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.692952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.692962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.693022 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.693290 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.694072 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.694798 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.695113 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.695192 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.697461 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:54281: BlockingRecv error: recv got EOF from 127.31.214.66:54281 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 06:23:57.697875 9173 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:23:57.765867 9176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:23:57.766127 9176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:23:57.766155 9176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:23:57.767333 9176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:23:57.767376 9176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:23:57.767409 9176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:23:57.768857 9176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/krb5kdc/oryx_127.31.214.67.keytab
--principal=oryx/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:38757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=42871
--tserver_master_addrs=127.31.214.126:43267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:23:57.769143 9176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:23:57.769349 9176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:23:57.775019 9185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.774987 9183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:23:57.774957 9182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:23:57.775238 9176 server_base.cc:1034] running on GCE node
I20260928 06:23:57.775518 9176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:23:57.775823 9176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:23:57.776984 9176 hybrid_clock.cc:648] HybridClock initialized: now 1790576637776947 us; error 58 us; skew 500 ppm
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576637, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:23:57.779603 9176 init.cc:377] Logged in from keytab as oryx/127.31.214.67@KRBTEST.COM (short username oryx)
I20260928 06:23:57.780079 9176 webserver.cc:460] Webserver started at http://127.31.214.67:42871/ using document root <none> and password file <none>
I20260928 06:23:57.780344 9176 fs_manager.cc:359] Metadata directory not provided
I20260928 06:23:57.780431 9176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:23:57.781893 9176 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:23:57.782662 9192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.782933 9176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:23:57.783048 9176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "d88647e989284ab2a1a479c38ed643a9"
format_stamp: "Formatted at 2026-09-28 06:23:57 on dist-test-slave-q03f"
server_key: "610bbb8174d21104d7b27b239073ddc4"
server_key_iv: "31d04dd40aacff69e71937406edf1b65"
server_key_version: "encryptionkey@0"
I20260928 06:23:57.783424 9176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:23:57.793550 9176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:23:57.795708 9176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:23:57.795898 9176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:23:57.796155 9176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:23:57.796492 9176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:23:57.796550 9176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.796587 9176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:23:57.796617 9176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:23:57.803017 9176 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:38757
I20260928 06:23:57.803092 9305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:38757 every 8 connection(s)
I20260928 06:23:57.803493 9176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:23:57.803954 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 9176
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:23:57.813092 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:57.804513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:44763 (local address 127.31.214.126:43267)
0928 06:23:57.804574 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.804577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.804824 (+ 247us) server_negotiation.cc:407] Connection header received
0928 06:23:57.805688 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.805689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.805699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.805750 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.807031 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.807666 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.809339 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.809689 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.809764 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.813031 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:44763: BlockingRecv error: recv got EOF from 127.31.214.67:44763 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:23:57.813243 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.804637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.31.214.126:43267)
0928 06:23:57.804828 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.804830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.804840 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:23:57.804870 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.804871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.804884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.804955 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.805854 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.806452 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.807556 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.807855 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.807962 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.809588 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:23:57.809593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:23:57.809593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:23:57.809607 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:23:57.811580 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:57.811982 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:57.811986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:57.811986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:57.812098 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:23:57.812338 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:23:57.812340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:23:57.812340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:23:57.812682 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:23:57.812838 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 06:23:57.812961 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:23:57.813081 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260928 06:23:57.813467 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.804480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:44763)
0928 06:23:57.804684 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.804740 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.805551 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.805853 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.805858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.806632 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.806923 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.806931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.807801 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.807802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.809237 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.809242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.809844 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.809845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.810013 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.810531 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.810540 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.812964 (+ 2424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
W20260928 06:23:57.813715 9306 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:43267 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.214.126:43267: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:57.818575 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.814019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:55023)
0928 06:23:57.814058 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.814070 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.814122 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.814276 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.814277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.814415 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.814461 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.814463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.815194 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.815195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.815773 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.815776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.816235 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.816235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.816284 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.816717 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.816721 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.818514 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 06:23:57.818758 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:57.814022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:55023 (local address 127.31.214.126:43267)
0928 06:23:57.814058 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.814060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.814108 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:23:57.814186 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.814187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.814197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.814253 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.814549 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.815122 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.815878 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.816151 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.816231 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.818678 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:55023: BlockingRecv error: recv got EOF from 127.31.214.67:55023 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:57 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:57.823051 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:57.818898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:50261)
0928 06:23:57.818935 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:57.818946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:57.818986 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:57.819107 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:57.819108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:57.819226 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:57.819250 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.819252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.819883 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.819885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.820377 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:57.820379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.820736 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.820737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:57.820788 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.821187 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:57.821191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:57.822966 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 06:23:57.823099 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:57.818931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:50261 (local address 127.31.214.126:43267)
0928 06:23:57.819004 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:57.819006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:57.819014 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:23:57.819037 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:57.819038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:57.819047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:57.819096 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:57.819344 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.819801 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.820468 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:57.820661 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:57.820739 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:57.823037 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:50261: BlockingRecv error: recv got EOF from 127.31.214.67:50261 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 06:23:57.823187 9306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 06:23:58 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:58 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:58.577538 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:58.572084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:41415)
0928 06:23:58.572224 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:58.572238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:58.572312 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:58.572467 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:58.572468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:58.572630 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:58.572683 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:58.572684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.573542 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.573544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:58.574172 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:58.574175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.574573 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.574574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:58.574628 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:58.575177 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:58.575182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:58.577456 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":54,"threads_started":1}
W20260928 06:23:58.577704 9312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:58.572191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:41415 (local address 127.31.214.126:43267)
0928 06:23:58.572344 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:58.572346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:58.572355 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:23:58.572378 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:58.572379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:58.572391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:58.572448 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:58.572833 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.573420 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.574257 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.574474 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.574544 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:58.577614 (+ 3070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:41415: BlockingRecv error: recv got EOF from 127.31.214.65:41415 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
Sep 28 06:23:58 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:58 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:58.703685 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:58.698388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:33721)
0928 06:23:58.698638 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:58.698651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:58.698776 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:58.698958 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:58.698959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:58.699145 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:58.699194 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:58.699196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.699960 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.699961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:58.700535 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:58.700537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.700941 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.700942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:58.701010 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:58.701577 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:58.701582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:58.703581 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":135,"threads_started":1}
W20260928 06:23:58.703742 9312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:58.698535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:33721 (local address 127.31.214.126:43267)
0928 06:23:58.698633 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:58.698636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:58.698731 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:23:58.698855 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:58.698857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:58.698871 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:58.698950 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:58.699305 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.699861 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.700643 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.700848 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.700940 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:58.703664 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:33721: BlockingRecv error: recv got EOF from 127.31.214.66:33721 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 06:23:58 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:58 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:58.828686 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:58.823624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:53997)
0928 06:23:58.823778 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:58.823795 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:58.823856 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:58.824035 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:58.824036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:58.824201 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:58.824254 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:58.824255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.824892 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.824893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:58.825477 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:58.825479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.826118 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.826119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:58.826197 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:58.826779 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:58.826785 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:58.828618 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
W20260928 06:23:58.828785 9312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:58.823729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:53997 (local address 127.31.214.126:43267)
0928 06:23:58.823790 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:58.823793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:58.823841 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:23:58.823946 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:58.823947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:58.823961 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:58.824008 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:58.824320 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.824822 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.825599 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:58.825934 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:58.826053 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:58.828715 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:53997: BlockingRecv error: recv got EOF from 127.31.214.67:53997 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 06:23:59 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:59 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:59.583271 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:59.578133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:37095)
0928 06:23:59.578259 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:59.578273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:59.578356 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:59.578660 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:59.578662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:59.578838 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:59.578890 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:59.578892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.579615 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.579616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:59.580241 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:59.580243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.580635 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.580636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:59.580687 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:59.581185 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:59.581190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:59.583208 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
W20260928 06:23:59.583415 9316 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:59.578209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:37095 (local address 127.31.214.126:43267)
0928 06:23:59.578373 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:59.578375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:59.578386 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:23:59.578412 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:59.578413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:59.578425 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:59.578540 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:59.578997 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.579533 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.580327 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.580542 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.580609 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:59.583301 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:37095: BlockingRecv error: recv got EOF from 127.31.214.65:37095 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
Sep 28 06:23:59 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:59 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:59.709697 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:59.704396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:44953)
0928 06:23:59.704632 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:59.704646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:59.704717 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:59.704931 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:59.704932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:59.705126 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:59.705203 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:59.705205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.706024 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.706025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:59.706595 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:59.706596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.707010 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.707012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:59.707067 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:59.707732 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:59.707737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:59.709634 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":136,"threads_started":1}
W20260928 06:23:59.709793 9316 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:59.704551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:44953 (local address 127.31.214.126:43267)
0928 06:23:59.704644 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:59.704646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:59.704722 (+ 76us) server_negotiation.cc:407] Connection header received
0928 06:23:59.704792 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:59.704793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:59.704804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:59.704901 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:59.705326 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.705930 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.706690 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.706913 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.706985 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:59.709743 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:44953: BlockingRecv error: recv got EOF from 127.31.214.66:44953 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 06:23:59 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:23:59 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:23:59.834149 9318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:23:59.829255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:55745)
0928 06:23:59.829421 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:23:59.829434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:23:59.829498 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:23:59.829737 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:23:59.829739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:23:59.829979 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:23:59.830028 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:59.830030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.830744 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.830745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:59.831348 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:23:59.831350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.831739 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.831741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:23:59.831803 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:59.832293 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:23:59.832297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:23:59.834046 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
W20260928 06:23:59.834205 9316 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:23:59.829507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:55745 (local address 127.31.214.126:43267)
0928 06:23:59.829590 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:23:59.829592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:23:59.829605 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:23:59.829627 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:23:59.829628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:23:59.829639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:23:59.829732 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:23:59.830122 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.830641 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.831458 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:23:59.831652 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:23:59.831729 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:23:59.834138 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:55745: BlockingRecv error: recv got EOF from 127.31.214.67:55745 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 06:24:00 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:00 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:00.589066 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:00.583832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:41865)
0928 06:24:00.584024 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:00.584037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:00.584106 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:00.584328 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:00.584329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:00.584518 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:00.584572 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:00.584575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.585398 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.585399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:00.586080 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:00.586120 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.586487 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.586488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:00.586534 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:00.587085 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:00.587090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:00.588993 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
W20260928 06:24:00.589224 9320 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:00.583930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:41865 (local address 127.31.214.126:43267)
0928 06:24:00.584070 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:00.584072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:00.584092 (+ 20us) server_negotiation.cc:407] Connection header received
0928 06:24:00.584181 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:00.584181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:00.584194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:00.584298 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:00.584673 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.585271 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.586201 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.586419 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.586479 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:00.589129 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:41865: BlockingRecv error: recv got EOF from 127.31.214.65:41865 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
Sep 28 06:24:00 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:00 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:00.716229 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:00.710474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:37585)
0928 06:24:00.710632 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:00.710646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:00.710717 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:00.710929 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:00.710930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:00.711124 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:00.711177 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:00.711179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.712112 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.712114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:00.712709 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:00.712711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.713123 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.713124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:00.713181 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:00.713817 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:00.713821 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:00.716127 (+ 2306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260928 06:24:00.716295 9320 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:00.710599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:37585 (local address 127.31.214.126:43267)
0928 06:24:00.710668 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:00.710670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:00.710705 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:24:00.710795 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:00.710796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:00.710809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:00.710896 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:00.711405 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.712018 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.712817 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.713041 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.713126 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:00.716226 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:37585: BlockingRecv error: recv got EOF from 127.31.214.66:37585 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 06:24:00 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:00 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:00.840193 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:00.834842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:40731)
0928 06:24:00.835025 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:00.835040 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:00.835117 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:00.835298 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:00.835299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:00.835487 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:00.835536 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:00.835538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.836347 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.836349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:00.836920 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:00.836922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.837316 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.837317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:00.837382 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:00.837907 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:00.837913 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:00.840081 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
W20260928 06:24:00.840233 9320 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:00.834991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:40731 (local address 127.31.214.126:43267)
0928 06:24:00.835067 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:00.835069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:00.835097 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:24:00.835197 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:00.835198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:00.835214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:00.835285 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:00.835628 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.836239 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.837015 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:00.837219 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:00.837313 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:00.840167 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:40731: BlockingRecv error: recv got EOF from 127.31.214.67:40731 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 06:24:01 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:01 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:01.603277 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:01.589598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:49205)
0928 06:24:01.589747 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:01.589760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:01.589837 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:01.590047 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:01.590048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:01.590237 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:01.590287 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:01.590289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.591056 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.591058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:01.591602 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:01.591604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.591987 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.591988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:01.592051 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:01.600598 (+ 8547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:01.600606 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:01.603194 (+ 2588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
W20260928 06:24:01.603433 9324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:01.589638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:49205 (local address 127.31.214.126:43267)
0928 06:24:01.589813 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:01.589815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:01.589826 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:24:01.589900 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:01.589901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:01.589915 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:01.590023 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:01.590391 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.590959 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.591699 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.591906 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.591987 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:01.603321 (+ 11334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:49205: BlockingRecv error: recv got EOF from 127.31.214.65:49205 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
Sep 28 06:24:01 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:01 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:01.722395 9325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:01.716958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:49393)
0928 06:24:01.717228 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:01.717241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:01.717351 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:01.717504 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:01.717505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:01.717698 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:01.717750 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:01.717753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.718579 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.718581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:01.719133 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:01.719135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.719622 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.719623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:01.719688 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:01.720275 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:01.720280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:01.722286 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
W20260928 06:24:01.722457 9324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:01.717136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:49393 (local address 127.31.214.126:43267)
0928 06:24:01.717197 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:01.717200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:01.717336 (+ 136us) server_negotiation.cc:407] Connection header received
0928 06:24:01.717422 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:01.717423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:01.717438 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:01.717500 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:01.717850 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.718468 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.719249 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.719505 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.719609 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:01.722384 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:49393: BlockingRecv error: recv got EOF from 127.31.214.66:49393 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 06:24:01 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:01 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:01.846485 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:01.840783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:58243)
0928 06:24:01.841029 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:01.841044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:01.841130 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:01.841289 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:01.841290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:01.841481 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:01.841532 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:01.841534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.842335 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.842337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:01.842911 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:01.842913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.843308 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.843309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:01.843504 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:01.844148 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:01.844154 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:01.846418 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
W20260928 06:24:01.846666 9324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:01.840950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:58243 (local address 127.31.214.126:43267)
0928 06:24:01.841052 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:01.841061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:01.841106 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:24:01.841189 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:01.841190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:01.841205 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:01.841279 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:01.841633 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.842246 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.843019 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:01.843217 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:01.843297 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:01.846565 (+ 3268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:58243: BlockingRecv error: recv got EOF from 127.31.214.67:58243 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 06:24:02 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:02 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:02.609340 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:02.603974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:57607)
0928 06:24:02.604174 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:02.604209 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:02.604268 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:02.604423 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:02.604424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:02.604590 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:02.604640 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:02.604642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.605496 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.605497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:02.606216 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:02.606219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.606633 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.606634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:02.606765 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:02.607300 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:02.607305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:02.609273 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":111,"threads_started":1}
W20260928 06:24:02.609544 9328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:02.604105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:57607 (local address 127.31.214.126:43267)
0928 06:24:02.604249 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:02.604251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:02.604261 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:24:02.604319 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:02.604320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:02.604331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:02.604406 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:02.604713 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.605389 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.606305 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.606546 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.606611 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:02.609433 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:57607: BlockingRecv error: recv got EOF from 127.31.214.65:57607 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
Sep 28 06:24:02 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:02 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:02.728708 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:02.723186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:52555)
0928 06:24:02.723407 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:02.723419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:02.723511 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:02.723761 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:02.723762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:02.723935 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:02.723993 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:02.723995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.724816 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.724818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:02.725551 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:02.725553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.725982 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.725983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:02.726040 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:02.726574 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:02.726580 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:02.728641 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
W20260928 06:24:02.728972 9328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:02.723438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:52555 (local address 127.31.214.126:43267)
0928 06:24:02.723508 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:02.723510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:02.723521 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:24:02.723579 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:02.723580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:02.723593 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:02.723649 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:02.724104 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.724748 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.725654 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.725879 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.725989 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:02.728901 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:52555: BlockingRecv error: recv got EOF from 127.31.214.66:52555 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 06:24:02 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:02 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:02.852383 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:02.847029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:49295)
0928 06:24:02.847170 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:02.847185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:02.847260 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:02.847456 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:02.847457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:02.847644 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:02.847698 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:02.847700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.848441 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.848442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:02.849074 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:02.849077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.849450 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.849450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:02.849495 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:02.850100 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:02.850106 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:02.852315 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
W20260928 06:24:02.852535 9328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:02.847124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:49295 (local address 127.31.214.126:43267)
0928 06:24:02.847179 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:02.847182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:02.847246 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:24:02.847337 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:02.847339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:02.847356 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:02.847420 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:02.847806 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.848342 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.849163 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:02.849373 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:02.849431 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:02.852433 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:49295: BlockingRecv error: recv got EOF from 127.31.214.67:49295 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 06:24:03 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:03 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:03.615167 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:03.609994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:59805)
0928 06:24:03.610162 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:03.610174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:03.610238 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:03.610386 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:03.610387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:03.610553 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:03.610616 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:03.610619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.611322 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.611323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:03.611888 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:03.611891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.612254 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.612255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:03.612313 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:03.612774 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:03.612805 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:03.615099 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
W20260928 06:24:03.615337 9332 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:03.610016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:59805 (local address 127.31.214.126:43267)
0928 06:24:03.610173 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:03.610175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:03.610222 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:24:03.610284 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:03.610285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:03.610295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:03.610363 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:03.610688 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.611239 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.611978 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.612177 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.612234 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:03.615215 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:59805: BlockingRecv error: recv got EOF from 127.31.214.65:59805 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
Sep 28 06:24:03 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:03 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:03.734975 9333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:03.729284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:40943)
0928 06:24:03.729519 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:03.729533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:03.729717 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:03.729908 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:03.729909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:03.730138 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:03.730201 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:03.730203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.730951 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.730953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:03.731507 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:03.731509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.731955 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.731956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:03.732016 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:03.732612 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:03.732617 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:03.734872 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
W20260928 06:24:03.735049 9332 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:03.729380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:40943 (local address 127.31.214.126:43267)
0928 06:24:03.729501 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:03.729504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:03.729620 (+ 116us) server_negotiation.cc:407] Connection header received
0928 06:24:03.729810 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:03.729811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:03.729827 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:03.729891 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:03.730300 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.730836 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.731610 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.731843 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.731953 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:03.734981 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:40943: BlockingRecv error: recv got EOF from 127.31.214.66:40943 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 06:24:03 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:03 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:03.858287 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:03.853061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:40485)
0928 06:24:03.853277 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:03.853291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:03.853348 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:03.853542 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:03.853543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:03.853760 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:03.853811 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:03.853813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.854598 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.854599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:03.855239 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:03.855241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.855612 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.855613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:03.855658 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:03.856330 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:03.856335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:03.858219 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":121,"threads_started":1}
W20260928 06:24:03.858510 9332 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:03.853223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:40485 (local address 127.31.214.126:43267)
0928 06:24:03.853359 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:03.853361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:03.853371 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:24:03.853396 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:03.853397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:03.853409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:03.853484 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:03.854000 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.854512 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.855332 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:03.855536 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:03.855596 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:03.858427 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:40485: BlockingRecv error: recv got EOF from 127.31.214.67:40485 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 06:24:04 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:04 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:04.623054 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:04.615847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:49297)
0928 06:24:04.616115 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:04.616130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:04.616203 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:04.616577 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:04.616579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:04.616811 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:04.616882 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:04.616885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.618084 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.618085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:04.619082 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:04.619085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.619694 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.619695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:04.619771 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:04.620383 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:04.620389 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:04.622931 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":206,"threads_started":1}
W20260928 06:24:04.623064 9336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:04.616190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:49297 (local address 127.31.214.126:43267)
0928 06:24:04.616397 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:04.616400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:04.616412 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:24:04.616442 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:04.616443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:04.616460 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:04.616552 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:04.616982 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.617999 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.619198 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.619595 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.619680 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:04.623007 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:49297: BlockingRecv error: recv got EOF from 127.31.214.65:49297 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
Sep 28 06:24:04 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:04 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:04.740847 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:04.735571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:41073)
0928 06:24:04.735798 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:04.735815 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:04.735899 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:04.736103 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:04.736104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:04.736282 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:04.736336 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:04.736337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.737023 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.737023 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:04.737583 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:04.737585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.738062 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.738067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:04.738120 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:04.738704 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:04.738708 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:04.740770 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
W20260928 06:24:04.740958 9336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:04.735850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:41073 (local address 127.31.214.126:43267)
0928 06:24:04.735936 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:04.735937 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:04.735947 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:24:04.736033 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:04.736034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:04.736044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:04.736094 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:04.736402 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.736940 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.737670 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.737906 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.738040 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:04.740884 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:41073: BlockingRecv error: recv got EOF from 127.31.214.66:41073 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 06:24:04 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:04 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:04.864282 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:04.858972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:48477)
0928 06:24:04.859218 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:04.859229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:04.859299 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:04.859451 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:04.859452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:04.859614 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:04.859754 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:04.859757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.860485 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.860486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:04.861111 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:04.861114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.861667 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.861668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:04.861729 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:04.862295 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:04.862300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:04.864213 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":128,"threads_started":1}
W20260928 06:24:04.864464 9336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:04.859032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:48477 (local address 127.31.214.126:43267)
0928 06:24:04.859107 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:04.859110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:04.859340 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:24:04.859366 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:04.859367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:04.859387 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:04.859434 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:04.859826 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.860406 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.861194 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:04.861586 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:04.861645 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:04.864334 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:48477: BlockingRecv error: recv got EOF from 127.31.214.67:48477 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 06:24:05 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:05 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:05.629035 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:05.623644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:44667)
0928 06:24:05.623765 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:05.623778 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:05.623836 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:05.624089 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:05.624090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:05.624298 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:05.624363 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:05.624366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.625093 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.625094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:05.625703 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:05.625705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.626315 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.626322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:05.626379 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:05.626884 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:05.626889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:05.628973 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
W20260928 06:24:05.629205 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:05.623732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:44667 (local address 127.31.214.126:43267)
0928 06:24:05.623920 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:05.623922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:05.623931 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:24:05.623960 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:05.623961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:05.623975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:05.624054 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:05.624460 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.624996 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.625891 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.626193 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.626263 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:05.629091 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:44667: BlockingRecv error: recv got EOF from 127.31.214.65:44667 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
Sep 28 06:24:05 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:05 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:05.746764 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:05.741477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:51887)
0928 06:24:05.741685 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:05.741698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:05.741813 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:05.742008 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:05.742008 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:05.742170 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:05.742222 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:05.742223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.743064 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.743066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:05.743616 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:05.743618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.744005 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.744006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:05.744059 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:05.744607 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:05.744612 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:05.746702 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
W20260928 06:24:05.746869 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:05.741572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:51887 (local address 127.31.214.126:43267)
0928 06:24:05.741687 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:05.741689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:05.741793 (+ 104us) server_negotiation.cc:407] Connection header received
0928 06:24:05.741881 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:05.741881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:05.741892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:05.741937 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:05.742289 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.742976 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.743700 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.743903 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.743961 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:05.746799 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:51887: BlockingRecv error: recv got EOF from 127.31.214.66:51887 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 06:24:05 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:05 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:05.870371 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:05.864820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:33659)
0928 06:24:05.864966 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:05.864983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:05.865056 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:05.865177 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:05.865178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:05.865330 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:05.865388 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:05.865389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.866247 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.866249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:05.866974 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:05.866979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.867466 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.867467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:05.867520 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:05.868049 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:05.868055 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:05.870309 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
W20260928 06:24:05.870528 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:05.864898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:33659 (local address 127.31.214.126:43267)
0928 06:24:05.864960 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:05.864962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:05.865069 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:24:05.865099 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:05.865099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:05.865112 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:05.865172 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:05.865454 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.866115 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.867068 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:05.867377 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:05.867438 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:05.870430 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:33659: BlockingRecv error: recv got EOF from 127.31.214.67:33659 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 06:24:06 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:06 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:06.641885 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:06.629573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:47415)
0928 06:24:06.629729 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:06.629740 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:06.629793 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:06.630022 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:06.630024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:06.630191 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:06.630242 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:06.630244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.630932 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.630934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:06.631483 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:06.631485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.631841 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.631842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:06.631887 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:06.639853 (+ 7966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:06.639859 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:06.641815 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
W20260928 06:24:06.642076 9344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:06.629661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:47415 (local address 127.31.214.126:43267)
0928 06:24:06.629803 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:06.629805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:06.629812 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:24:06.629836 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:06.629837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:06.629848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:06.629925 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:06.630321 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.630850 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.631566 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.631766 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.631823 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:06.641979 (+ 10156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:47415: BlockingRecv error: recv got EOF from 127.31.214.65:47415 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
Sep 28 06:24:06 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:06 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:06.752625 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:06.747335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:48617)
0928 06:24:06.747476 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:06.747489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:06.747577 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:06.747733 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:06.747734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:06.747896 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:06.747948 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:06.747950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.748600 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.748601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:06.749149 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:06.749151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.749541 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.749542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:06.749598 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:06.750221 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:06.750228 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:06.752508 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
W20260928 06:24:06.752666 9344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:06.747379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:48617 (local address 127.31.214.126:43267)
0928 06:24:06.747451 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:06.747453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:06.747556 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:24:06.747649 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:06.747650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:06.747665 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:06.747725 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:06.748014 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.748522 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.749241 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.749445 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.749505 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:06.752598 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:48617: BlockingRecv error: recv got EOF from 127.31.214.66:48617 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 06:24:06 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:06 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:06.876111 9346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:06.870942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:60755)
0928 06:24:06.871105 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:06.871115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:06.871166 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:06.871330 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:06.871331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:06.871536 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:06.871637 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:06.871640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.872341 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.872342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:06.872920 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:06.872922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.873286 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.873287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:06.873341 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:06.873882 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:06.873886 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:06.876001 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
W20260928 06:24:06.876147 9344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:06.871076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:60755 (local address 127.31.214.126:43267)
0928 06:24:06.871148 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:06.871150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:06.871164 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:24:06.871222 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:06.871223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:06.871237 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:06.871313 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:06.871757 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.872256 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.873023 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:06.873213 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:06.873287 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:06.876079 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:60755: BlockingRecv error: recv got EOF from 127.31.214.67:60755 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 06:24:07 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:07 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:07.647625 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:07.642476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:59041)
0928 06:24:07.642654 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:07.642666 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:07.642780 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:07.642961 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:07.642962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:07.643149 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:07.643204 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:07.643206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.643874 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.643875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:07.644467 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:07.644469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.644812 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.644813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:07.644861 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:07.645456 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:07.645461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:07.647552 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
W20260928 06:24:07.647732 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:07.642555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:59041 (local address 127.31.214.126:43267)
0928 06:24:07.642694 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:07.642696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:07.642763 (+ 67us) server_negotiation.cc:407] Connection header received
0928 06:24:07.642855 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:07.642855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:07.642866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:07.642930 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:07.643272 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.643789 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.644545 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.644741 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.644793 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:07.647654 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:59041: BlockingRecv error: recv got EOF from 127.31.214.65:59041 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
Sep 28 06:24:07 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:07 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:07.758522 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:07.753191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:59737)
0928 06:24:07.753335 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:07.753354 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:07.753440 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:07.753574 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:07.753575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:07.753760 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:07.753810 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:07.753812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.754585 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.754587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:07.755169 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:07.755171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.755576 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.755577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:07.755640 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:07.756228 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:07.756233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:07.758453 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20260928 06:24:07.758623 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:07.753353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:59737 (local address 127.31.214.126:43267)
0928 06:24:07.753430 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:07.753432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:07.753442 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:24:07.753505 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:07.753506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:07.753519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:07.753572 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:07.753907 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.754478 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.755268 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.755496 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.755577 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:07.758571 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:59737: BlockingRecv error: recv got EOF from 127.31.214.66:59737 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:24:07 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:07 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:07.882128 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:07.876682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:45647)
0928 06:24:07.876832 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:07.876843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:07.876917 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:07.877066 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:07.877067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:07.877252 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:07.877303 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:07.877305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.878237 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.878239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:07.878867 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:07.878870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.879342 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.879343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:07.879412 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:07.880016 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:07.880021 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:07.882017 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
W20260928 06:24:07.882180 9348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:07.876859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:45647 (local address 127.31.214.126:43267)
0928 06:24:07.876932 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:07.876934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:07.876942 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:24:07.876970 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:07.876971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:07.876986 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:07.877059 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:07.877398 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.878138 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.879000 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:07.879238 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:07.879343 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:07.882106 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:45647: BlockingRecv error: recv got EOF from 127.31.214.67:45647 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 06:24:08 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:08 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:08.653723 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:08.648243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:46395)
0928 06:24:08.648396 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:08.648408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:08.648499 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:08.648756 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:08.648758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:08.648933 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:08.649005 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:08.649007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.649730 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.649731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:08.650306 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:08.650308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.650659 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.650660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:08.650706 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:08.651264 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:08.651271 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:08.653599 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260928 06:24:08.653792 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:08.648374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:46395 (local address 127.31.214.126:43267)
0928 06:24:08.648517 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:08.648519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:08.648530 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:24:08.648556 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:08.648557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:08.648568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:08.648644 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:08.649100 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.649633 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.650386 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.650582 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.650645 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:08.653705 (+ 3060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:46395: BlockingRecv error: recv got EOF from 127.31.214.65:46395 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
Sep 28 06:24:08 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:08 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:08.764811 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:08.759272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:59113)
0928 06:24:08.759460 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:08.759472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:08.759524 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:08.759672 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:08.759673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:08.759873 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:08.759923 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:08.759924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.760641 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.760642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:08.761338 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:08.761341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.762021 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.762022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:08.762097 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:08.762760 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:08.762765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:08.764749 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":118,"threads_started":1}
W20260928 06:24:08.765012 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:08.759433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:59113 (local address 127.31.214.126:43267)
0928 06:24:08.759498 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:08.759500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:08.759557 (+ 57us) server_negotiation.cc:407] Connection header received
0928 06:24:08.759586 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:08.759587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:08.759599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:08.759645 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:08.760014 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.760560 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.761467 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.761880 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.762018 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:08.764941 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:59113: BlockingRecv error: recv got EOF from 127.31.214.66:59113 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:24:08 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:08 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:08.888532 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:08.882730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:36749)
0928 06:24:08.882958 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:08.882970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:08.883036 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:08.883195 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:08.883195 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:08.883402 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:08.883451 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:08.883452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.884365 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.884366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:08.884998 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:08.885001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.885480 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.885482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:08.885549 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:08.886178 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:08.886183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:08.888401 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":150,"threads_started":1}
W20260928 06:24:08.888576 9352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:08.882838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:36749 (local address 127.31.214.126:43267)
0928 06:24:08.882893 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:08.882895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:08.883059 (+ 164us) server_negotiation.cc:407] Connection header received
0928 06:24:08.883093 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:08.883094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:08.883110 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:08.883177 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:08.883537 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.884257 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.885134 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:08.885358 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:08.885458 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:08.888498 (+ 3040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:36749: BlockingRecv error: recv got EOF from 127.31.214.67:36749 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 06:24:09 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:09 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:09.659415 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:09.654379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:59617)
0928 06:24:09.654558 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:09.654569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:09.654623 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:09.654838 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:09.654839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:09.655013 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:09.655062 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:09.655064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.655861 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.655862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:09.656399 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:09.656402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.656823 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.656824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:09.656893 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:09.657414 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:09.657419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:09.659352 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
W20260928 06:24:09.659585 9356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:09.654511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:59617 (local address 127.31.214.126:43267)
0928 06:24:09.654677 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:09.654679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:09.654689 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:24:09.654715 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:09.654716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:09.654726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:09.654801 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:09.655135 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.655723 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.656512 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.656731 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.656817 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:09.659492 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:59617: BlockingRecv error: recv got EOF from 127.31.214.65:59617 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
Sep 28 06:24:09 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:09 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:09.770597 9357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:09.765373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:60667)
0928 06:24:09.765538 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:09.765550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:09.765644 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:09.765856 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:09.765857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:09.766071 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:09.766119 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:09.766121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.766840 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.766841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:09.767387 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:09.767389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.767764 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.767765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:09.767820 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:09.768370 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:09.768375 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:09.770526 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
W20260928 06:24:09.770746 9356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:09.765537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:60667 (local address 127.31.214.126:43267)
0928 06:24:09.765627 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:09.765629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:09.765641 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:24:09.765711 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:09.765712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:09.765730 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:09.765843 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:09.766208 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.766728 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.767479 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.767686 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.767775 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:09.770652 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:60667: BlockingRecv error: recv got EOF from 127.31.214.66:60667 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 06:24:09 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:09 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:09.894192 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:09.889092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:38221)
0928 06:24:09.889252 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:09.889298 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:09.889360 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:09.889501 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:09.889502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:09.889672 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:09.889718 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:09.889720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.890618 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.890619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:09.891185 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:09.891188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.891567 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.891568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:09.891615 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:09.892140 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:09.892144 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:09.894069 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
W20260928 06:24:09.894225 9356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:09.889203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:38221 (local address 127.31.214.126:43267)
0928 06:24:09.889254 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:09.889256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:09.889363 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:24:09.889421 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:09.889422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:09.889435 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:09.889482 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:09.889831 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.890530 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.891292 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:09.891496 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:09.891552 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:09.894154 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:38221: BlockingRecv error: recv got EOF from 127.31.214.67:38221 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 06:24:10 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:10 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:10.665527 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:10.660054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:40573)
0928 06:24:10.660170 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:10.660183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:10.660245 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:10.660385 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:10.660385 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:10.660550 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:10.660600 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:10.660601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.661397 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.661398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:10.662181 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:10.662185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.662557 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.662558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:10.662607 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:10.663222 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:10.663227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:10.665460 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
W20260928 06:24:10.665720 9360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:10.660099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:40573 (local address 127.31.214.126:43267)
0928 06:24:10.660221 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:10.660223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:10.660233 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:24:10.660303 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:10.660303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:10.660314 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:10.660379 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:10.660659 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.661293 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.662282 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.662484 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.662542 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:10.665617 (+ 3075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:40573: BlockingRecv error: recv got EOF from 127.31.214.65:40573 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
Sep 28 06:24:10 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:10 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:10.776677 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:10.771208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:59927)
0928 06:24:10.771385 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:10.771406 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:10.771472 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:10.771615 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:10.771616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:10.771857 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:10.771927 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:10.771929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.772809 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.772810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:10.773411 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:10.773413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.773880 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.773881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:10.773980 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:10.774507 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:10.774511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:10.776574 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":126,"threads_started":1}
W20260928 06:24:10.776741 9360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:10.771307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:59927 (local address 127.31.214.126:43267)
0928 06:24:10.771353 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:10.771355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:10.771472 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:24:10.771522 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:10.771522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:10.771533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:10.771587 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:10.772040 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.772678 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.773504 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.773774 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.773845 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:10.776680 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:59927: BlockingRecv error: recv got EOF from 127.31.214.66:59927 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:24:10 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:10 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:10.899765 9362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:10.894886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:56975)
0928 06:24:10.895058 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:10.895074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:10.895150 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:10.895351 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:10.895352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:10.895535 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:10.895589 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:10.895590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.896333 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.896334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:10.896940 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:10.896942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.897294 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.897295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:10.897341 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:10.897818 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:10.897823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:10.899707 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
W20260928 06:24:10.899924 9360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:10.894952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:56975 (local address 127.31.214.126:43267)
0928 06:24:10.895040 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:10.895042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:10.895142 (+ 100us) server_negotiation.cc:407] Connection header received
0928 06:24:10.895195 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:10.895196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:10.895208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:10.895261 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:10.895684 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.896238 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.897026 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:10.897213 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:10.897270 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:10.899825 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:56975: BlockingRecv error: recv got EOF from 127.31.214.67:56975 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 06:24:11 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:11 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:11.677407 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:11.666115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:57679)
0928 06:24:11.666245 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:11.666257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:11.666320 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:11.666451 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:11.666452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:11.666615 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:11.666664 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:11.666666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.667418 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.667418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:11.668002 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:11.668004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.668400 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.668401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:11.668442 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:11.675034 (+ 6592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:11.675041 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:11.677333 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
W20260928 06:24:11.677577 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:11.666188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:57679 (local address 127.31.214.126:43267)
0928 06:24:11.666321 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:11.666323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:11.666331 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:24:11.666367 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:11.666367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:11.666377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:11.666444 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:11.666759 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.667340 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.668086 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.668326 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.668389 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:11.677484 (+ 9095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:57679: BlockingRecv error: recv got EOF from 127.31.214.65:57679 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
Sep 28 06:24:11 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:11 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:11.782860 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:11.777339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:60669)
0928 06:24:11.777490 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:11.777505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:11.777574 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:11.777741 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:11.777742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:11.777902 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:11.777985 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:11.777987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.778923 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.778924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:11.779516 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:11.779518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.779989 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.779990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:11.780045 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:11.780624 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:11.780636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:11.782796 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
W20260928 06:24:11.782994 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:11.777438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:60669 (local address 127.31.214.126:43267)
0928 06:24:11.777489 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:11.777491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:11.777576 (+ 85us) server_negotiation.cc:407] Connection header received
0928 06:24:11.777621 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:11.777622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:11.777633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:11.777681 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:11.778109 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.778782 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.779669 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.779888 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.779959 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:11.782911 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:60669: BlockingRecv error: recv got EOF from 127.31.214.66:60669 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 06:24:11 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:11 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.67@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:11.905309 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:11.900333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.67:38143)
0928 06:24:11.900504 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:11.900514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:11.900579 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:11.900754 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:11.900755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:11.900935 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:11.900987 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:11.900989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.901739 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.901741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:11.902382 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:11.902385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.902745 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.902746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:11.902798 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:11.903376 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:11.903381 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:11.905236 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
W20260928 06:24:11.905436 9364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:11.900426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:38143 (local address 127.31.214.126:43267)
0928 06:24:11.900492 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:11.900494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:11.900578 (+ 84us) server_negotiation.cc:407] Connection header received
0928 06:24:11.900644 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:11.900645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:11.900658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:11.900727 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:11.901053 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.901600 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.902463 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:11.902653 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:11.902712 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:11.905350 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.67:38143: BlockingRecv error: recv got EOF from 127.31.214.67:38143 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 06:24:12 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:12 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.65@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:12.683677 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:12.678018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.65:60227)
0928 06:24:12.678187 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:12.678201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:12.678285 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:12.678541 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:12.678542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:12.678733 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:12.678785 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:12.678788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.679655 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.679657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:12.680352 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:12.680355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.680731 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.680732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:12.680798 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:12.681348 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:12.681353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:12.683502 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":57,"threads_started":1}
W20260928 06:24:12.683669 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:12.678127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:60227 (local address 127.31.214.126:43267)
0928 06:24:12.678329 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:12.678331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:12.678345 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:24:12.678397 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:12.678398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:12.678412 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:12.678487 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:12.678982 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.679541 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.680446 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.680652 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.680732 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:12.683598 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.65:60227: BlockingRecv error: recv got EOF from 127.31.214.65:60227 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":125,"threads_started":1}
Sep 28 06:24:12 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:24:12 dist-test-slave-q03f krb5kdc[8309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.214.66@KRBTEST.COM for oryx/127.31.214.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:24:12.789059 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:12.783492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43267 (local address 127.31.214.66:39321)
0928 06:24:12.783684 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:12.783699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:12.783796 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:12.784007 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:12.784008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:12.784222 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:12.784272 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:12.784274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.785078 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.785079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:12.785617 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:12.785619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.786185 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.786186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:12.786247 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:12.786818 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:12.786824 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:12.788997 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.214.126:43267: Server oryx/127.31.214.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":100,"threads_started":1}
W20260928 06:24:12.789233 9368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:12.783621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:39321 (local address 127.31.214.126:43267)
0928 06:24:12.783706 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:12.783708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:12.783776 (+ 68us) server_negotiation.cc:407] Connection header received
0928 06:24:12.783874 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:12.783875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:12.783889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:12.783995 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:12.784370 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.784980 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.785820 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.786104 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.786187 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:12.789161 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.214.66:39321: BlockingRecv error: recv got EOF from 127.31.214.66:39321 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:24:12.808558 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 8893
I20260928 06:24:12.813786 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 9044
I20260928 06:24:12.818789 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 9176
I20260928 06:24:12.823709 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 8822
2026-09-28T06:24:12Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17553 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T06:24:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:24:12Z Disabled control of system clock
I20260928 06:24:12.853068 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:39763
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:39763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:24:12.942641 9373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:12.942870 9373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:12.942891 9373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:12.944449 9373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:24:12.944500 9373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:12.944514 9373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:12.944526 9373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:24:12.944538 9373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:24:12.946069 9373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:39763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:39763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:12.946349 9373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:12.946673 9373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:12.952163 9378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:12.952176 9379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:12.952165 9381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:12.952571 9373 server_base.cc:1034] running on GCE node
I20260928 06:24:12.952997 9373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:12.953306 9373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:12.954478 9373 hybrid_clock.cc:648] HybridClock initialized: now 1790576652954466 us; error 50 us; skew 500 ppm
I20260928 06:24:12.956036 9373 webserver.cc:460] Webserver started at http://127.31.214.126:40241/ using document root <none> and password file <none>
I20260928 06:24:12.956310 9373 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:12.956387 9373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:12.956545 9373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:12.957691 9373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "adcbbe1755b648699c158599a1a8bf18"
format_stamp: "Formatted at 2026-09-28 06:24:12 on dist-test-slave-q03f"
server_key: "034092f38bb69c9614b5016c466eb5c5"
server_key_iv: "c053055ffddfe479092563f2b0b43679"
server_key_version: "encryptionkey@0"
I20260928 06:24:12.958112 9373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "adcbbe1755b648699c158599a1a8bf18"
format_stamp: "Formatted at 2026-09-28 06:24:12 on dist-test-slave-q03f"
server_key: "034092f38bb69c9614b5016c466eb5c5"
server_key_iv: "c053055ffddfe479092563f2b0b43679"
server_key_version: "encryptionkey@0"
I20260928 06:24:12.959697 9373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:24:12.960600 9387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:12.960841 9373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:24:12.960956 9373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "adcbbe1755b648699c158599a1a8bf18"
format_stamp: "Formatted at 2026-09-28 06:24:12 on dist-test-slave-q03f"
server_key: "034092f38bb69c9614b5016c466eb5c5"
server_key_iv: "c053055ffddfe479092563f2b0b43679"
server_key_version: "encryptionkey@0"
I20260928 06:24:12.961040 9373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:12.977142 9373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:12.977514 9373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:12.977685 9373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:12.981832 9373 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:39763
I20260928 06:24:12.981920 9439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:39763 every 8 connection(s)
I20260928 06:24:12.982311 9373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:24:12.983127 9440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:24:12.985936 9440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18: Bootstrap starting.
I20260928 06:24:12.986670 9440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18: Neither blocks nor log segments found. Creating new log.
I20260928 06:24:12.986991 9440 log.cc:824] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18: Log is configured to *not* fsync() on all Append() calls
I20260928 06:24:12.987872 9440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18: No bootstrap required, opened a new log
I20260928 06:24:12.988574 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 9373
I20260928 06:24:12.988677 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:24:12.988896 32601 external_mini_cluster.cc:1428] Setting key 296ab8d9a19cb6bc3e9f2b466c449fef
I20260928 06:24:12.990482 9440 raft_consensus.cc:348] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbbe1755b648699c158599a1a8bf18" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 39763 } }
I20260928 06:24:12.990778 9440 raft_consensus.cc:374] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:24:12.990833 9440 raft_consensus.cc:729] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcbbe1755b648699c158599a1a8bf18, State: Initialized, Role: FOLLOWER
I20260928 06:24:12.991055 9440 consensus_queue.cc:260] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbbe1755b648699c158599a1a8bf18" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 39763 } }
I20260928 06:24:12.991184 9440 raft_consensus.cc:388] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:24:12.991238 9440 raft_consensus.cc:482] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:24:12.991298 9440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:24:12.991973 9440 raft_consensus.cc:504] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbbe1755b648699c158599a1a8bf18" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 39763 } }
I20260928 06:24:12.992106 9440 leader_election.cc:302] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adcbbe1755b648699c158599a1a8bf18; no voters:
I20260928 06:24:12.992322 9440 leader_election.cc:288] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:24:12.992359 9445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:24:12.992527 9445 raft_consensus.cc:686] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [term 1 LEADER]: Becoming Leader. State: Replica: adcbbe1755b648699c158599a1a8bf18, State: Running, Role: LEADER
I20260928 06:24:12.992678 9445 consensus_queue.cc:237] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbbe1755b648699c158599a1a8bf18" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 39763 } }
I20260928 06:24:12.992691 9440 sys_catalog.cc:564] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:24:12.993244 9447 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adcbbe1755b648699c158599a1a8bf18. Latest consensus state: current_term: 1 leader_uuid: "adcbbe1755b648699c158599a1a8bf18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbbe1755b648699c158599a1a8bf18" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 39763 } } }
I20260928 06:24:12.993337 9447 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [sys.catalog]: This master's current role is: LEADER
I20260928 06:24:12.993518 9446 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adcbbe1755b648699c158599a1a8bf18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcbbe1755b648699c158599a1a8bf18" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 39763 } } }
I20260928 06:24:12.993587 9446 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18 [sys.catalog]: This master's current role is: LEADER
I20260928 06:24:12.993927 9451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:24:12.994547 9451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:24:12.995685 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:12.989518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.31.214.126:39763)
0928 06:24:12.989742 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:12.989749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:12.989777 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:24:12.990405 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:12.990412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:12.990476 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:12.990600 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:12.990916 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.992244 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.993308 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:12.994005 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:12.994136 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:12.994940 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:12.994945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:12.994945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:24:12.995145 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:12.995180 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:24:12.995191 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:12.995232 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:12.995395 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:24:12.995486 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260928 06:24:12.996475 9451 catalog_manager.cc:1347] Generated new cluster ID: cc82134f6a4a480c99c250928f6b904b
I20260928 06:24:12.996529 9451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:24:13.014722 9451 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:24:13.015523 9451 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:24:13.020347 9451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adcbbe1755b648699c158599a1a8bf18: Generated new TSK 0
I20260928 06:24:13.020591 9451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:24:13.027016 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:39763
--builtin_ntp_servers=127.31.214.84:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:24:13.115257 9464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:13.115535 9464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:13.115585 9464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:13.116842 9464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:13.116915 9464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:13.116961 9464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:24:13.118467 9464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:39763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:13.118755 9464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:13.119087 9464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:13.125461 9469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.125521 9472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:13.125816 9464 server_base.cc:1034] running on GCE node
W20260928 06:24:13.125521 9470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:13.126103 9464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:13.126381 9464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:13.127550 9464 hybrid_clock.cc:648] HybridClock initialized: now 1790576653127522 us; error 44 us; skew 500 ppm
I20260928 06:24:13.129099 9464 webserver.cc:460] Webserver started at http://127.31.214.65:39579/ using document root <none> and password file <none>
I20260928 06:24:13.129364 9464 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:13.129426 9464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:13.129550 9464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:13.131004 9464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "550025e56e384410aa32d98292f802c0"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "205bb9801a009e6a42fbf1b84fe0c1a2"
server_key_iv: "67ab5e02ad119395672375568a94ad94"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.131469 9464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "550025e56e384410aa32d98292f802c0"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "205bb9801a009e6a42fbf1b84fe0c1a2"
server_key_iv: "67ab5e02ad119395672375568a94ad94"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.132944 9464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:24:13.133924 9478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.134210 9464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:24:13.134337 9464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "550025e56e384410aa32d98292f802c0"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "205bb9801a009e6a42fbf1b84fe0c1a2"
server_key_iv: "67ab5e02ad119395672375568a94ad94"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.134397 9464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:13.160677 9464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:13.160993 9464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:13.161106 9464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:13.161324 9464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:24:13.161708 9464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:24:13.161748 9464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.161772 9464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:24:13.161823 9464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.168326 9464 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:46057
I20260928 06:24:13.168406 9591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:46057 every 8 connection(s)
I20260928 06:24:13.168781 9464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:24:13.172657 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 9464
I20260928 06:24:13.172739 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:24:13.172971 32601 external_mini_cluster.cc:1428] Setting key 0a7193aa302ab44068d1db9265caeb88
I20260928 06:24:13.174160 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:39763
--builtin_ntp_servers=127.31.214.84:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:24:13.174988 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.169647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:53405 (local address 127.31.214.126:39763)
0928 06:24:13.169726 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:13.169729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:13.170033 (+ 304us) server_negotiation.cc:407] Connection header received
0928 06:24:13.170705 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:13.170707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:13.170720 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:13.170796 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:13.171525 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.172060 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.173309 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.173544 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.173642 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:13.174721 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:13.174722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:13.174722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:24:13.174727 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:13.174743 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:24:13.174747 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:13.174801 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:13.174880 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 06:24:13.174908 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:24:13.175251 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.169633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:39763 (local address 127.31.214.65:53405)
0928 06:24:13.169877 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:13.169914 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:13.170502 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:13.170891 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:13.170895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:13.170904 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:13.171421 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:13.171426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.172164 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.172165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:13.173166 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:13.173170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.173663 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.173664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:13.173867 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:13.174564 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:24:13.174564 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:13.174595 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:24:13.174622 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:24:13.174807 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:24:13.174812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:24:13.174815 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:24:13.174885 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:24:13.174998 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":116,"threads_started":1}
I20260928 06:24:13.175822 9592 heartbeater.cc:347] Connected to a master server at 127.31.214.126:39763
I20260928 06:24:13.175945 9592 heartbeater.cc:464] Registering TS with master...
I20260928 06:24:13.176143 9592 heartbeater.cc:511] Master 127.31.214.126:39763 requested a full tablet report, sending...
I20260928 06:24:13.176705 9404 ts_manager.cc:194] Registered new tserver with Master: 550025e56e384410aa32d98292f802c0 (127.31.214.65:46057)
I20260928 06:24:13.177382 9404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.65:53405
W20260928 06:24:13.257481 9595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:13.257697 9595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:13.257718 9595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:13.258932 9595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:13.258978 9595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:13.259003 9595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:24:13.260406 9595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:39763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:13.260624 9595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:13.260874 9595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:13.266187 9603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.266209 9601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.266209 9600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:13.266475 9595 server_base.cc:1034] running on GCE node
I20260928 06:24:13.266742 9595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:13.266990 9595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:13.268136 9595 hybrid_clock.cc:648] HybridClock initialized: now 1790576653268123 us; error 33 us; skew 500 ppm
I20260928 06:24:13.269557 9595 webserver.cc:460] Webserver started at http://127.31.214.66:37067/ using document root <none> and password file <none>
I20260928 06:24:13.270038 9595 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:13.270100 9595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:13.270259 9595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:13.271550 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "7f2f5a531ffc4c13b2fa722b26ae7071"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "13e1695a704f949ef9d88cb668fd0e99"
server_key_iv: "c40b432abe465e6062f4d3074b623602"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.271930 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "7f2f5a531ffc4c13b2fa722b26ae7071"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "13e1695a704f949ef9d88cb668fd0e99"
server_key_iv: "c40b432abe465e6062f4d3074b623602"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.273303 9595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:24:13.274151 9609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.274374 9595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:24:13.274492 9595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "7f2f5a531ffc4c13b2fa722b26ae7071"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "13e1695a704f949ef9d88cb668fd0e99"
server_key_iv: "c40b432abe465e6062f4d3074b623602"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.274574 9595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:13.287505 9595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:13.287861 9595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:13.288033 9595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:13.288305 9595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:24:13.288678 9595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:24:13.288739 9595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.288808 9595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:24:13.288841 9595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.295884 9595 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:34383
I20260928 06:24:13.295928 9722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:34383 every 8 connection(s)
I20260928 06:24:13.296386 9595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:24:13.299829 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 9595
I20260928 06:24:13.299952 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:24:13.300185 32601 external_mini_cluster.cc:1428] Setting key 39cb43705a65beb4d3f2a69c42d724b3
I20260928 06:24:13.301215 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:39763
--builtin_ntp_servers=127.31.214.84:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:24:13.302595 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.297332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:52343 (local address 127.31.214.126:39763)
0928 06:24:13.297386 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:13.297388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:13.297636 (+ 248us) server_negotiation.cc:407] Connection header received
0928 06:24:13.298206 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:13.298208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:13.298219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:13.298287 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:13.299076 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.299660 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.300975 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.301178 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.301257 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:13.302292 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:13.302293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:13.302293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:24:13.302298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:13.302319 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:24:13.302323 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:13.302380 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:13.302486 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 06:24:13.302515 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:24:13.302894 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.297270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:39763 (local address 127.31.214.66:52343)
0928 06:24:13.297470 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:13.297513 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:13.298083 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:13.298375 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:13.298378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:13.298387 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:13.298981 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:13.298986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.299790 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.299791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:13.300895 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:13.300899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.301264 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.301265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:13.301439 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:13.302125 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:24:13.302125 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:13.302157 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:24:13.302183 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:24:13.302394 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:24:13.302399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:24:13.302402 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:24:13.302491 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:24:13.302697 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260928 06:24:13.303496 9723 heartbeater.cc:347] Connected to a master server at 127.31.214.126:39763
I20260928 06:24:13.303635 9723 heartbeater.cc:464] Registering TS with master...
I20260928 06:24:13.303952 9723 heartbeater.cc:511] Master 127.31.214.126:39763 requested a full tablet report, sending...
I20260928 06:24:13.304543 9404 ts_manager.cc:194] Registered new tserver with Master: 7f2f5a531ffc4c13b2fa722b26ae7071 (127.31.214.66:34383)
I20260928 06:24:13.305033 9404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.66:52343
W20260928 06:24:13.385922 9726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:13.386207 9726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:13.386230 9726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:13.387475 9726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:13.387516 9726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:13.387542 9726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:24:13.388942 9726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:39763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:13.389199 9726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:13.389506 9726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:13.395113 9731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.395113 9734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.395181 9732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:13.395341 9726 server_base.cc:1034] running on GCE node
I20260928 06:24:13.395581 9726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:13.395880 9726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:13.397063 9726 hybrid_clock.cc:648] HybridClock initialized: now 1790576653397040 us; error 43 us; skew 500 ppm
I20260928 06:24:13.398645 9726 webserver.cc:460] Webserver started at http://127.31.214.67:37263/ using document root <none> and password file <none>
I20260928 06:24:13.398973 9726 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:13.399055 9726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:13.399237 9726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:13.400447 9726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "673fd4b59dfb4682942a1253eebfbc12"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "f4008fe8d22065c6375ff5da5ff00db5"
server_key_iv: "3519b7d76e426e9a974f3c604bcb1cfa"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.400859 9726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "673fd4b59dfb4682942a1253eebfbc12"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "f4008fe8d22065c6375ff5da5ff00db5"
server_key_iv: "3519b7d76e426e9a974f3c604bcb1cfa"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.402436 9726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 06:24:13.403313 9740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.403524 9726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:24:13.403640 9726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "673fd4b59dfb4682942a1253eebfbc12"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "f4008fe8d22065c6375ff5da5ff00db5"
server_key_iv: "3519b7d76e426e9a974f3c604bcb1cfa"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.403754 9726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:13.422462 9726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:13.422833 9726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:13.423004 9726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:13.423272 9726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:24:13.423635 9726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:24:13.423692 9726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.423815 9726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:24:13.423846 9726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.430219 9726 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:41699
I20260928 06:24:13.430279 9853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:41699 every 8 connection(s)
I20260928 06:24:13.430639 9726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:24:13.436157 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.431373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:45929 (local address 127.31.214.126:39763)
0928 06:24:13.431445 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:13.431447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:13.431665 (+ 218us) server_negotiation.cc:407] Connection header received
0928 06:24:13.432256 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:13.432257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:13.432269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:13.432325 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:13.433011 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.433584 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.434618 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.434864 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.434928 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:13.435908 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:13.435910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:13.435910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:24:13.435914 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:13.435930 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:24:13.435934 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:13.435986 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:13.436072 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 06:24:13.436093 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:24:13.436429 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.431356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:39763 (local address 127.31.214.67:45929)
0928 06:24:13.431552 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:13.431589 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:13.432156 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:13.432384 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:13.432387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:13.432396 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:13.432918 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:13.432922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.433666 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.433667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:13.434532 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:13.434535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:13.434943 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:13.434944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:13.435055 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:13.435727 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:24:13.435727 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:13.435758 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:24:13.435799 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:24:13.435997 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:24:13.436001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:24:13.436004 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:24:13.436058 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:24:13.436146 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260928 06:24:13.437067 9854 heartbeater.cc:347] Connected to a master server at 127.31.214.126:39763
I20260928 06:24:13.437191 9854 heartbeater.cc:464] Registering TS with master...
I20260928 06:24:13.437383 9854 heartbeater.cc:511] Master 127.31.214.126:39763 requested a full tablet report, sending...
I20260928 06:24:13.437705 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 9726
I20260928 06:24:13.437842 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:24:13.437906 9404 ts_manager.cc:194] Registered new tserver with Master: 673fd4b59dfb4682942a1253eebfbc12 (127.31.214.67:41699)
I20260928 06:24:13.438134 32601 external_mini_cluster.cc:1428] Setting key de2aa5c2f80a4fec1d75dff075da279f
I20260928 06:24:13.438318 9404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.67:45929
I20260928 06:24:13.439163 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:24:13.441136 9444 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:13.440710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.31.214.126:39763)
0928 06:24:13.440815 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:13.440817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:13.440869 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:24:13.440931 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:13.440932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:13.440940 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:13.440988 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:13.441088 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56560: BlockingRecv error: recv got EOF from 127.0.0.1:56560 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 06:24:13.441246 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 9464
I20260928 06:24:13.445858 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 9595
I20260928 06:24:13.450671 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 9726
I20260928 06:24:13.455588 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 9373
2026-09-28T06:24:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (631 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T06:24:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:24:13Z Disabled control of system clock
I20260928 06:24:13.484774 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:40319
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:40319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:24:13.568488 9865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:13.568831 9865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:13.568858 9865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:13.570255 9865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:24:13.570308 9865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:13.570323 9865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:13.570338 9865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:24:13.570351 9865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:24:13.571964 9865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:40319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:40319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:13.572271 9865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:13.572625 9865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:13.578150 9873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.578155 9871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.578255 9870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:13.578445 9865 server_base.cc:1034] running on GCE node
I20260928 06:24:13.578768 9865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:13.579047 9865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:13.580231 9865 hybrid_clock.cc:648] HybridClock initialized: now 1790576653580215 us; error 50 us; skew 500 ppm
I20260928 06:24:13.581770 9865 webserver.cc:460] Webserver started at http://127.31.214.126:42795/ using document root <none> and password file <none>
I20260928 06:24:13.582087 9865 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:13.582162 9865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:13.582312 9865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:13.583532 9865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "f76b834abaf6495482195896f513f69b"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "76033702b030680e64781789a9bb1474"
server_key_iv: "8b2955c724ed3928d776aaa2f4fcc92a"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.583940 9865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "f76b834abaf6495482195896f513f69b"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "76033702b030680e64781789a9bb1474"
server_key_iv: "8b2955c724ed3928d776aaa2f4fcc92a"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.585333 9865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:24:13.586351 9879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.586555 9865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:24:13.586746 9865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "f76b834abaf6495482195896f513f69b"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "76033702b030680e64781789a9bb1474"
server_key_iv: "8b2955c724ed3928d776aaa2f4fcc92a"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.586829 9865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:13.590478 9865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:13.590772 9865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:13.590914 9865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:13.594985 9865 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:40319
I20260928 06:24:13.595016 9931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:40319 every 8 connection(s)
I20260928 06:24:13.595357 9865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:24:13.596086 9932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:24:13.598964 9932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b: Bootstrap starting.
I20260928 06:24:13.599555 9932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b: Neither blocks nor log segments found. Creating new log.
I20260928 06:24:13.599617 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 9865
I20260928 06:24:13.599769 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:24:13.599923 9932 log.cc:824] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:24:13.599990 32601 external_mini_cluster.cc:1428] Setting key 5c291d289a1a42244e523da383913e5e
I20260928 06:24:13.600761 9932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b: No bootstrap required, opened a new log
I20260928 06:24:13.602793 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.600564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.31.214.126:40319)
0928 06:24:13.600775 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:13.600782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:13.600824 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:24:13.601261 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:13.601268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:13.601304 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:13.601416 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:13.602195 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:13.602198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:13.602199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:24:13.602203 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:13.602251 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:24:13.602262 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:13.602349 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:13.602427 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:24:13.602513 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260928 06:24:13.603011 9932 raft_consensus.cc:348] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76b834abaf6495482195896f513f69b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 40319 } }
I20260928 06:24:13.603189 9932 raft_consensus.cc:374] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:24:13.603225 9932 raft_consensus.cc:729] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76b834abaf6495482195896f513f69b, State: Initialized, Role: FOLLOWER
I20260928 06:24:13.603353 9932 consensus_queue.cc:260] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76b834abaf6495482195896f513f69b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 40319 } }
I20260928 06:24:13.603433 9932 raft_consensus.cc:388] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:24:13.603480 9932 raft_consensus.cc:482] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:24:13.603531 9932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:24:13.604422 9932 raft_consensus.cc:504] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76b834abaf6495482195896f513f69b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 40319 } }
I20260928 06:24:13.604543 9932 leader_election.cc:302] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f76b834abaf6495482195896f513f69b; no voters:
I20260928 06:24:13.604727 9932 leader_election.cc:288] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:24:13.604779 9937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:24:13.604900 9937 raft_consensus.cc:686] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [term 1 LEADER]: Becoming Leader. State: Replica: f76b834abaf6495482195896f513f69b, State: Running, Role: LEADER
I20260928 06:24:13.604991 9937 consensus_queue.cc:237] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76b834abaf6495482195896f513f69b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 40319 } }
I20260928 06:24:13.605019 9932 sys_catalog.cc:564] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:24:13.605331 9939 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f76b834abaf6495482195896f513f69b. Latest consensus state: current_term: 1 leader_uuid: "f76b834abaf6495482195896f513f69b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76b834abaf6495482195896f513f69b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 40319 } } }
I20260928 06:24:13.605458 9939 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [sys.catalog]: This master's current role is: LEADER
I20260928 06:24:13.605710 9942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:24:13.606263 9942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:24:13.606812 9938 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f76b834abaf6495482195896f513f69b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76b834abaf6495482195896f513f69b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 40319 } } }
I20260928 06:24:13.606905 9938 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b [sys.catalog]: This master's current role is: LEADER
I20260928 06:24:13.607879 9942 catalog_manager.cc:1347] Generated new cluster ID: aced307c5eb24651822fbf7bd8c6d4a6
I20260928 06:24:13.607934 9942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:24:13.617434 9942 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:24:13.617636 9942 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:24:13.625006 9942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f76b834abaf6495482195896f513f69b: Generated new TSK 0
I20260928 06:24:13.625252 9942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:24:13.627347 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:40319
--builtin_ntp_servers=127.31.214.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:24:13.722433 9956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:13.722692 9956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:13.722745 9956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:13.724140 9956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:13.724220 9956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:13.724270 9956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:24:13.725798 9956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:40319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:13.726150 9956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:13.726503 9956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:13.732661 9961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.732661 9962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.732864 9964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:13.733057 9956 server_base.cc:1034] running on GCE node
I20260928 06:24:13.733289 9956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:13.733582 9956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:13.734812 9956 hybrid_clock.cc:648] HybridClock initialized: now 1790576653734791 us; error 37 us; skew 500 ppm
I20260928 06:24:13.736272 9956 webserver.cc:460] Webserver started at http://127.31.214.65:46789/ using document root <none> and password file <none>
I20260928 06:24:13.736542 9956 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:13.736620 9956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:13.736850 9956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:13.738165 9956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "37e356b26c33403a8182650da6e62b1f"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "821be89595901ad188c31c18dd55bb55"
server_key_iv: "7775b2424cb0f3e3cfe3642ad69f1865"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.738579 9956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "37e356b26c33403a8182650da6e62b1f"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "821be89595901ad188c31c18dd55bb55"
server_key_iv: "7775b2424cb0f3e3cfe3642ad69f1865"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.740052 9956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:24:13.740911 9970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.741127 9956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:24:13.741256 9956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "37e356b26c33403a8182650da6e62b1f"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "821be89595901ad188c31c18dd55bb55"
server_key_iv: "7775b2424cb0f3e3cfe3642ad69f1865"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.741343 9956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:13.745072 9956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:13.745378 9956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:13.745536 9956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:13.745774 9956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:24:13.746181 9956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:24:13.746227 9956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.746271 9956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:24:13.746299 9956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.752279 9956 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:37225
I20260928 06:24:13.752362 10083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:37225 every 8 connection(s)
I20260928 06:24:13.752720 9956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:24:13.753688 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 9956
I20260928 06:24:13.753787 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:24:13.754021 32601 external_mini_cluster.cc:1428] Setting key a831c2bfbfba30fba2e93632f77f917f
I20260928 06:24:13.755147 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:40319
--builtin_ntp_servers=127.31.214.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 06:24:13.756553 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.753490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:41571 (local address 127.31.214.126:40319)
0928 06:24:13.753571 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:13.753574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:13.753852 (+ 278us) server_negotiation.cc:407] Connection header received
0928 06:24:13.754575 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:13.754577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:13.754584 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:13.754656 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:13.756253 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:13.756255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:13.756255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:24:13.756259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:13.756275 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:24:13.756279 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:13.756333 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:13.756436 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:24:13.756450 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:24:13.756753 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.753483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:40319 (local address 127.31.214.65:41571)
0928 06:24:13.753705 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:13.753754 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:13.754466 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:13.754778 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:13.754782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:13.754792 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:13.756067 (+ 1275us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:24:13.756068 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:13.756106 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:24:13.756151 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:24:13.756351 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:24:13.756357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:24:13.756361 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:24:13.756439 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:24:13.756507 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260928 06:24:13.757229 10084 heartbeater.cc:347] Connected to a master server at 127.31.214.126:40319
I20260928 06:24:13.757345 10084 heartbeater.cc:464] Registering TS with master...
I20260928 06:24:13.757514 10084 heartbeater.cc:511] Master 127.31.214.126:40319 requested a full tablet report, sending...
I20260928 06:24:13.758083 9896 ts_manager.cc:194] Registered new tserver with Master: 37e356b26c33403a8182650da6e62b1f (127.31.214.65:37225)
W20260928 06:24:13.848124 10087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:13.848356 10087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:13.848381 10087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:13.849707 10087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:13.849754 10087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:13.849784 10087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:24:13.851387 10087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:40319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:13.851686 10087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:13.852038 10087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:13.857797 10093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.857877 10092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.857798 10095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:13.858083 10087 server_base.cc:1034] running on GCE node
I20260928 06:24:13.858368 10087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:13.858621 10087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:13.859773 10087 hybrid_clock.cc:648] HybridClock initialized: now 1790576653859753 us; error 39 us; skew 500 ppm
I20260928 06:24:13.861138 10087 webserver.cc:460] Webserver started at http://127.31.214.66:39023/ using document root <none> and password file <none>
I20260928 06:24:13.861430 10087 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:13.861506 10087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:13.861660 10087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:13.862869 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "badef90881ab42008bdfe37195f58bf2"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "95307c7684925b12c3bd79e323e0b7f0"
server_key_iv: "b1ca43567c67d76d07edf0cc4c6576a7"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.863252 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "badef90881ab42008bdfe37195f58bf2"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "95307c7684925b12c3bd79e323e0b7f0"
server_key_iv: "b1ca43567c67d76d07edf0cc4c6576a7"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.864622 10087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:24:13.865430 10101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.865675 10087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:24:13.865805 10087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "badef90881ab42008bdfe37195f58bf2"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "95307c7684925b12c3bd79e323e0b7f0"
server_key_iv: "b1ca43567c67d76d07edf0cc4c6576a7"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.865885 10087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:13.869287 10087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:13.869521 10087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:13.869616 10087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:13.869778 10087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:24:13.870177 10087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:24:13.870241 10087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.870283 10087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:24:13.870311 10087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.876672 10087 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:36583
I20260928 06:24:13.876722 10214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:36583 every 8 connection(s)
I20260928 06:24:13.877096 10087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:24:13.880543 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.877985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:58131 (local address 127.31.214.126:40319)
0928 06:24:13.878039 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:13.878042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:13.878321 (+ 279us) server_negotiation.cc:407] Connection header received
0928 06:24:13.878924 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:13.878925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:13.878933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:13.878985 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:13.880247 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:13.880248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:13.880248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:24:13.880253 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:13.880269 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:24:13.880274 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:13.880354 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:13.880450 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 06:24:13.880466 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:24:13.880774 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:13.877985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:40319 (local address 127.31.214.66:58131)
0928 06:24:13.878177 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:13.878217 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:13.878801 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:13.879068 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:13.879072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:13.879080 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:13.880081 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:24:13.880082 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:13.880111 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:24:13.880143 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:24:13.880356 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:24:13.880362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:24:13.880366 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:24:13.880479 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:24:13.880546 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260928 06:24:13.881055 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 10087
I20260928 06:24:13.881150 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:24:13.881304 10215 heartbeater.cc:347] Connected to a master server at 127.31.214.126:40319
I20260928 06:24:13.881363 32601 external_mini_cluster.cc:1428] Setting key bf1a565caeb87138e99753c909ca9dda
I20260928 06:24:13.881417 10215 heartbeater.cc:464] Registering TS with master...
I20260928 06:24:13.881569 10215 heartbeater.cc:511] Master 127.31.214.126:40319 requested a full tablet report, sending...
I20260928 06:24:13.882073 9896 ts_manager.cc:194] Registered new tserver with Master: badef90881ab42008bdfe37195f58bf2 (127.31.214.66:36583)
I20260928 06:24:13.882397 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:40319
--builtin_ntp_servers=127.31.214.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:24:13.969692 10218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:13.969913 10218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:13.969992 10218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:13.971220 10218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:13.971273 10218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:13.971305 10218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:24:13.972867 10218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:40319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:13.973153 10218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:13.973490 10218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:13.978991 10224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.979022 10226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:13.978991 10223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:13.979447 10218 server_base.cc:1034] running on GCE node
I20260928 06:24:13.979748 10218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:13.979975 10218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:13.981114 10218 hybrid_clock.cc:648] HybridClock initialized: now 1790576653981102 us; error 30 us; skew 500 ppm
I20260928 06:24:13.982688 10218 webserver.cc:460] Webserver started at http://127.31.214.67:42643/ using document root <none> and password file <none>
I20260928 06:24:13.982959 10218 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:13.983033 10218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:13.983211 10218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:13.984339 10218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "9ccd530c2ebb4dd6b4d369fe778d86ae"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "0166df9278f9fb7067cd505f5e626807"
server_key_iv: "01b907c03997f57f45bcf4cd6dfff190"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.984722 10218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "9ccd530c2ebb4dd6b4d369fe778d86ae"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "0166df9278f9fb7067cd505f5e626807"
server_key_iv: "01b907c03997f57f45bcf4cd6dfff190"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.986255 10218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:24:13.987042 10232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.987222 10218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:24:13.987330 10218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "9ccd530c2ebb4dd6b4d369fe778d86ae"
format_stamp: "Formatted at 2026-09-28 06:24:13 on dist-test-slave-q03f"
server_key: "0166df9278f9fb7067cd505f5e626807"
server_key_iv: "01b907c03997f57f45bcf4cd6dfff190"
server_key_version: "encryptionkey@0"
I20260928 06:24:13.987403 10218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:13.991178 10218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:13.991462 10218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:13.991607 10218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:13.991827 10218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:24:13.992141 10218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:24:13.992195 10218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.992235 10218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:24:13.992264 10218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:13.998510 10218 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:42637
I20260928 06:24:13.998976 10218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:24:14.000638 10345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:42637 every 8 connection(s)
I20260928 06:24:14.002298 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 10218
I20260928 06:24:14.002399 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:24:14.002633 32601 external_mini_cluster.cc:1428] Setting key 2b4cf5b852d3d15a4de77a757448422d
I20260928 06:24:14.006407 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:14.000192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:47335 (local address 127.31.214.126:40319)
0928 06:24:14.000394 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:14.000396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:14.003479 (+ 3083us) server_negotiation.cc:407] Connection header received
0928 06:24:14.004237 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:14.004239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:14.004248 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:14.004316 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:14.005932 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:14.005934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:14.005934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:24:14.005939 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:14.006013 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:24:14.006019 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:14.006112 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:14.006275 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:24:14.006293 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260928 06:24:14.006858 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:14.000031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:40319 (local address 127.31.214.67:47335)
0928 06:24:14.003327 (+ 3296us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:14.003375 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:14.004099 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:14.004444 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:14.004448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:14.004460 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:14.005736 (+ 1276us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:24:14.005737 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:14.005783 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:24:14.005825 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:24:14.006146 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:24:14.006154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:24:14.006157 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:24:14.006529 (+ 372us) client_negotiation.cc:240] Negotiation successful
0928 06:24:14.006614 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3221,"thread_start_us":3176,"threads_started":1}
I20260928 06:24:14.010162 10346 heartbeater.cc:347] Connected to a master server at 127.31.214.126:40319
I20260928 06:24:14.010298 10346 heartbeater.cc:464] Registering TS with master...
I20260928 06:24:14.010483 10346 heartbeater.cc:511] Master 127.31.214.126:40319 requested a full tablet report, sending...
I20260928 06:24:14.010929 9896 ts_manager.cc:194] Registered new tserver with Master: 9ccd530c2ebb4dd6b4d369fe778d86ae (127.31.214.67:42637)
I20260928 06:24:14.014742 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:24:14.016953 9936 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:24:14.016380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.31.214.126:40319)
0928 06:24:14.016435 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:14.016437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:14.016532 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:24:14.016635 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:14.016636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:14.016642 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:14.016698 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:14.016875 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34484: BlockingRecv error: recv got EOF from 127.0.0.1:34484 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:24:14.016983 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 9956
I20260928 06:24:14.023952 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 10087
I20260928 06:24:14.031507 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 10218
I20260928 06:24:14.037343 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 9865
2026-09-28T06:24:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (581 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:24:14 dist-test-slave-q03f krb5kdc[10356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:24:14 dist-test-slave-q03f krb5kdc[10356](info): set up 2 sockets
Sep 28 06:24:14 dist-test-slave-q03f krb5kdc[10356](info): commencing operation
krb5kdc: starting...
W20260928 06:24:15.150617 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:24:15 dist-test-slave-q03f krb5kdc[10356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:24:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:24:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:24:15.277441 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45391
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:45391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:24:15.362404 10369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:15.362681 10369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:15.362723 10369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:15.363947 10369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:24:15.364018 10369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:15.364044 10369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:15.364068 10369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:24:15.364096 10369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:24:15.365607 10369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:45391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:15.365904 10369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:15.366168 10369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:15.371917 10377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:15.371909 10375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:15.371939 10369 server_base.cc:1034] running on GCE node
W20260928 06:24:15.371909 10374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:15.372516 10369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:15.372932 10369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:15.374122 10369 hybrid_clock.cc:648] HybridClock initialized: now 1790576655374118 us; error 49 us; skew 500 ppm
Sep 28 06:24:15 dist-test-slave-q03f krb5kdc[10356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576655, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:24:15.376979 10369 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:24:15.377460 10369 webserver.cc:460] Webserver started at http://127.31.214.126:44723/ using document root <none> and password file <none>
I20260928 06:24:15.377730 10369 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:15.377808 10369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:15.378003 10369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:15.379182 10369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "2f62293a433f4490951a7e56d9a47cc3"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "9c3310c999ee5a5fbe7b0b8c0043a038"
server_key_iv: "80409840c207bc2d4b6ee1af7bb92fda"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.379578 10369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "2f62293a433f4490951a7e56d9a47cc3"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "9c3310c999ee5a5fbe7b0b8c0043a038"
server_key_iv: "80409840c207bc2d4b6ee1af7bb92fda"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.381140 10369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:24:15.382071 10384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.382323 10369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:24:15.382428 10369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "2f62293a433f4490951a7e56d9a47cc3"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "9c3310c999ee5a5fbe7b0b8c0043a038"
server_key_iv: "80409840c207bc2d4b6ee1af7bb92fda"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.382521 10369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:15.396052 10369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:15.398731 10369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:15.398938 10369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:15.403215 10369 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:45391
I20260928 06:24:15.403252 10436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:45391 every 8 connection(s)
I20260928 06:24:15.403638 10369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:24:15.404771 10437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:24:15.407306 10437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3: Bootstrap starting.
I20260928 06:24:15.407941 10437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3: Neither blocks nor log segments found. Creating new log.
I20260928 06:24:15.408249 10437 log.cc:824] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:24:15.409185 10437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3: No bootstrap required, opened a new log
I20260928 06:24:15.411104 10437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62293a433f4490951a7e56d9a47cc3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45391 } }
I20260928 06:24:15.411258 10437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:24:15.411281 10437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f62293a433f4490951a7e56d9a47cc3, State: Initialized, Role: FOLLOWER
I20260928 06:24:15.411458 10437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62293a433f4490951a7e56d9a47cc3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45391 } }
I20260928 06:24:15.411536 10437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:24:15.411590 10437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:24:15.411649 10437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:24:15.412551 10437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62293a433f4490951a7e56d9a47cc3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45391 } }
I20260928 06:24:15.412674 10437 leader_election.cc:302] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f62293a433f4490951a7e56d9a47cc3; no voters:
I20260928 06:24:15.412868 10437 leader_election.cc:288] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:24:15.412933 10440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:24:15.412952 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 10369
I20260928 06:24:15.413045 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:24:15.413098 10440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [term 1 LEADER]: Becoming Leader. State: Replica: 2f62293a433f4490951a7e56d9a47cc3, State: Running, Role: LEADER
I20260928 06:24:15.413153 10437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:24:15.413203 10440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62293a433f4490951a7e56d9a47cc3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45391 } }
I20260928 06:24:15.413280 32601 external_mini_cluster.cc:1428] Setting key b6193ae3b3c47075945121a62a698a12
I20260928 06:24:15.413548 10442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f62293a433f4490951a7e56d9a47cc3. Latest consensus state: current_term: 1 leader_uuid: "2f62293a433f4490951a7e56d9a47cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62293a433f4490951a7e56d9a47cc3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45391 } } }
I20260928 06:24:15.413561 10441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f62293a433f4490951a7e56d9a47cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f62293a433f4490951a7e56d9a47cc3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45391 } } }
I20260928 06:24:15.413717 10441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:24:15.413924 10442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:24:15.414273 10445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:24:15.414866 10445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:24:15.416549 10445 catalog_manager.cc:1347] Generated new cluster ID: ae3f9f27607a448cac6a631d1e013369
I20260928 06:24:15.416615 10445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:24:15 dist-test-slave-q03f krb5kdc[10356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:24:15.424283 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.414008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.31.214.126:45391)
0928 06:24:15.414175 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:15.414185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:15.414223 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:24:15.414359 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:15.414368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:15.414448 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:15.415497 (+ 1049us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:15.415594 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.416908 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.417696 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.418248 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.418386 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.420793 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:15.420797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:15.420798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:24:15.420816 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:15.422618 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.423005 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.423009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.423009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.423123 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.423355 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.423356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.423357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.423620 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:15.423809 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:15.423884 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:24:15.424039 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260928 06:24:15.448300 10445 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:24:15.448977 10445 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:24:15.455246 10445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3: Generated new TSK 0
I20260928 06:24:15.455504 10445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:24:15.486145 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45391
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:24:15.571954 10463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:15.572181 10463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:15.572207 10463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:15.573310 10463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:15.573345 10463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:15.573371 10463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:24:15.574923 10463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:15.575181 10463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:15.575376 10463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:15.580639 10469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:15.580832 10471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:15.580952 10463 server_base.cc:1034] running on GCE node
W20260928 06:24:15.580651 10468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:15.581260 10463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:15.581483 10463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:15.582828 10463 hybrid_clock.cc:648] HybridClock initialized: now 1790576655582797 us; error 45 us; skew 500 ppm
Sep 28 06:24:15 dist-test-slave-q03f krb5kdc[10356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576655, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:24:15.585635 10463 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:24:15.586191 10463 webserver.cc:460] Webserver started at http://127.31.214.65:46175/ using document root <none> and password file <none>
I20260928 06:24:15.586477 10463 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:15.586525 10463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:15.586675 10463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:15.587767 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "7b5540ed40544e809d7688a236c3ed1e"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "410baeb1825b87e185f23651f3331b6b"
server_key_iv: "44e821433e3dc2f04c1fc0f427630e61"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.588146 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "7b5540ed40544e809d7688a236c3ed1e"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "410baeb1825b87e185f23651f3331b6b"
server_key_iv: "44e821433e3dc2f04c1fc0f427630e61"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.589515 10463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:24:15.590400 10478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.590577 10463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:24:15.590683 10463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "7b5540ed40544e809d7688a236c3ed1e"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "410baeb1825b87e185f23651f3331b6b"
server_key_iv: "44e821433e3dc2f04c1fc0f427630e61"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.590767 10463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:15.606974 10463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:15.609543 10463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:15.609730 10463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:15.610008 10463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:24:15.610374 10463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:24:15.610430 10463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.610466 10463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:24:15.610507 10463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.616885 10463 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:46241
I20260928 06:24:15.617079 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:46241 every 8 connection(s)
I20260928 06:24:15.617349 10463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:24:15.621805 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 10463
I20260928 06:24:15.621943 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:24:15.622174 32601 external_mini_cluster.cc:1428] Setting key 6b21849ba871adcbafd81c7bd9193141
Sep 28 06:24:15 dist-test-slave-q03f krb5kdc[10356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576655, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:24:15.627976 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.618141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:38697 (local address 127.31.214.126:45391)
0928 06:24:15.618205 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:15.618207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:15.618489 (+ 282us) server_negotiation.cc:407] Connection header received
0928 06:24:15.619001 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:15.619002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:15.619016 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:15.619086 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:15.620110 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.620724 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.621912 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.622443 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.622543 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.624969 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:15.624971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:15.624972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:24:15.624981 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:15.626516 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.626956 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.626957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.626958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.627023 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.627288 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.627288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.627289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.627412 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:15.627488 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:15.627788 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 06:24:15.627880 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:24:15.628199 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.618141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45391 (local address 127.31.214.65:38697)
0928 06:24:15.618355 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:15.618396 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:15.618889 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:15.619152 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:15.619155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:15.619788 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:15.619997 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.620002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.620864 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.620865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.621783 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.621785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.622572 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.622573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.622723 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.623336 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:15.623345 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:15.624871 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:24:15.626631 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:24:15.626639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:24:15.626641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:24:15.626876 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:24:15.627089 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:24:15.627090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:24:15.627090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:24:15.627213 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:24:15.627479 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:24:15.627494 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:24:15.627646 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 06:24:15.627754 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:24:15.627939 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":140,"threads_started":1}
I20260928 06:24:15.628691 10592 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45391
I20260928 06:24:15.628813 10592 heartbeater.cc:464] Registering TS with master...
I20260928 06:24:15.629036 10592 heartbeater.cc:511] Master 127.31.214.126:45391 requested a full tablet report, sending...
I20260928 06:24:15.629628 10401 ts_manager.cc:194] Registered new tserver with Master: 7b5540ed40544e809d7688a236c3ed1e (127.31.214.65:46241)
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
I20260928 06:24:15.630353 10401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:38697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:24:15.651196 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45391
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:24:15.735921 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:15.736135 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:15.736159 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:15.737351 10597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:15.737399 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:15.737426 10597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:24:15.738961 10597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:15.739219 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:15.739461 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:15.744819 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:15.744872 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:15.744743 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:15.744966 10597 server_base.cc:1034] running on GCE node
I20260928 06:24:15.745227 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:15.745482 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:15.746661 10597 hybrid_clock.cc:648] HybridClock initialized: now 1790576655746647 us; error 35 us; skew 500 ppm
Sep 28 06:24:15 dist-test-slave-q03f krb5kdc[10356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576655, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:24:15.749457 10597 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:24:15.750078 10597 webserver.cc:460] Webserver started at http://127.31.214.66:46839/ using document root <none> and password file <none>
I20260928 06:24:15.750371 10597 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:15.750443 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:15.750602 10597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:15.751698 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "6d415e9d76dd4f85bc49eb10a4b2911a"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "97ea4dd42862e744a97070c4f02f381b"
server_key_iv: "382090e87e5963d3ea40a4da75965d8f"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.752076 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "6d415e9d76dd4f85bc49eb10a4b2911a"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "97ea4dd42862e744a97070c4f02f381b"
server_key_iv: "382090e87e5963d3ea40a4da75965d8f"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.753499 10597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:24:15.754408 10612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.754606 10597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:24:15.754709 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "6d415e9d76dd4f85bc49eb10a4b2911a"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "97ea4dd42862e744a97070c4f02f381b"
server_key_iv: "382090e87e5963d3ea40a4da75965d8f"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.754794 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:15.781870 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:15.784036 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:15.784212 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:15.784509 10597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:24:15.784889 10597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:24:15.784941 10597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.784984 10597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:24:15.785012 10597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.791213 10597 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:45979
I20260928 06:24:15.791267 10725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:45979 every 8 connection(s)
I20260928 06:24:15.791682 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:24:15.796866 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 10597
I20260928 06:24:15.796957 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:24:15.797195 32601 external_mini_cluster.cc:1428] Setting key bdc067fe0248cd6e835a5aeeda051231
Sep 28 06:24:15 dist-test-slave-q03f krb5kdc[10356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576655, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:24:15.803287 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.792541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45391 (local address 127.31.214.66:36977)
0928 06:24:15.792794 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:15.792848 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:15.793408 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:15.793651 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:15.793655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:15.794451 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:15.794660 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.794665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.795521 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.795522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.796392 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.796395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.796790 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.796792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.796964 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.797601 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:15.797611 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:15.799274 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:24:15.801554 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:24:15.801562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:24:15.801564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:24:15.801911 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:24:15.802194 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:24:15.802194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:24:15.802195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:24:15.802312 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:24:15.802726 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:24:15.802732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:24:15.802869 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 06:24:15.802956 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:24:15.803096 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":110,"threads_started":1}
I20260928 06:24:15.803297 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.792659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:36977 (local address 127.31.214.126:45391)
0928 06:24:15.792732 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:15.792735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:15.792935 (+ 200us) server_negotiation.cc:407] Connection header received
0928 06:24:15.793524 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:15.793525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:15.793537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:15.793598 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:15.794844 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.795391 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.796477 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.796674 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.796756 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.799412 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:15.799414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:15.799414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:24:15.799425 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:15.801456 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.802043 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.802045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.802045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.802119 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.802392 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.802393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.802394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.802586 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:15.802669 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:15.803029 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 06:24:15.803141 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:24:15.804003 10726 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45391
I20260928 06:24:15.804119 10726 heartbeater.cc:464] Registering TS with master...
I20260928 06:24:15.804371 10726 heartbeater.cc:511] Master 127.31.214.126:45391 requested a full tablet report, sending...
I20260928 06:24:15.804790 10401 ts_manager.cc:194] Registered new tserver with Master: 6d415e9d76dd4f85bc49eb10a4b2911a (127.31.214.66:45979)
I20260928 06:24:15.805162 10401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:36977
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:24:15.826128 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45391
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:24:15.911311 10731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:24:15.911561 10731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:24:15.911602 10731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:24:15.912945 10731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:24:15.913018 10731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:24:15.913069 10731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:24:15.914892 10731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:24:15.915158 10731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:24:15.915405 10731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:24:15.920742 10736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:15.920895 10737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:24:15.921036 10739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:24:15.921162 10731 server_base.cc:1034] running on GCE node
I20260928 06:24:15.921437 10731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:24:15.921732 10731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:24:15.922896 10731 hybrid_clock.cc:648] HybridClock initialized: now 1790576655922862 us; error 47 us; skew 500 ppm
Sep 28 06:24:15 dist-test-slave-q03f krb5kdc[10356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576655, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:24:15.925493 10731 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:24:15.926069 10731 webserver.cc:460] Webserver started at http://127.31.214.67:40723/ using document root <none> and password file <none>
I20260928 06:24:15.926316 10731 fs_manager.cc:359] Metadata directory not provided
I20260928 06:24:15.926388 10731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:24:15.926546 10731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:24:15.927642 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "87bf64a39a694b70ae53402ca3fc25cf"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "b5cd5c8451e1c3eb93d538d18ea85741"
server_key_iv: "31f4dd248d166d6fe541d886f65c0403"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.928022 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "87bf64a39a694b70ae53402ca3fc25cf"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "b5cd5c8451e1c3eb93d538d18ea85741"
server_key_iv: "31f4dd248d166d6fe541d886f65c0403"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.929474 10731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:24:15.930330 10746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.930575 10731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:24:15.930689 10731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "87bf64a39a694b70ae53402ca3fc25cf"
format_stamp: "Formatted at 2026-09-28 06:24:15 on dist-test-slave-q03f"
server_key: "b5cd5c8451e1c3eb93d538d18ea85741"
server_key_iv: "31f4dd248d166d6fe541d886f65c0403"
server_key_version: "encryptionkey@0"
I20260928 06:24:15.930773 10731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:24:15.945123 10731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:24:15.947496 10731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:24:15.947681 10731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:24:15.947942 10731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:24:15.948343 10731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:24:15.948393 10731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.948433 10731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:24:15.948457 10731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:24:15.954859 10731 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:34411
I20260928 06:24:15.955086 10859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:34411 every 8 connection(s)
I20260928 06:24:15.955334 10731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:24:15.961470 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 10731
I20260928 06:24:15.961617 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:24:15.961997 32601 external_mini_cluster.cc:1428] Setting key 9fe776ae7bcbe9c1b9ff12fba4827d6b
Sep 28 06:24:15 dist-test-slave-q03f krb5kdc[10356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576655, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:24:15.966774 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.956451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:54225 (local address 127.31.214.126:45391)
0928 06:24:15.956569 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:15.956572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:15.956801 (+ 229us) server_negotiation.cc:407] Connection header received
0928 06:24:15.957457 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:15.957458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:15.957471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:15.957554 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:15.958987 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.959499 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.960522 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.960798 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.960860 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.963808 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:15.963810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:15.963810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:24:15.963817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:15.965169 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.965646 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.965647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.965648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.965711 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.966028 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.966029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.966030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.966162 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:15.966241 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:15.966583 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 06:24:15.966683 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 06:24:15.966974 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.956448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45391 (local address 127.31.214.67:54225)
0928 06:24:15.956631 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:15.956704 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:15.957325 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:15.957834 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:15.957838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:15.958598 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:24:15.958863 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.958869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.959595 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.959596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.960438 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.960441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.960898 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.960899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.961046 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.961772 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:24:15.961784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:24:15.963657 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:24:15.965291 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:24:15.965299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:24:15.965301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:24:15.965543 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:24:15.965791 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:24:15.965792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:24:15.965792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:24:15.965912 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:24:15.966242 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:24:15.966250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:24:15.966401 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 06:24:15.966506 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:24:15.966704 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260928 06:24:15.967631 10860 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45391
I20260928 06:24:15.967761 10860 heartbeater.cc:464] Registering TS with master...
I20260928 06:24:15.967967 10860 heartbeater.cc:511] Master 127.31.214.126:45391 requested a full tablet report, sending...
I20260928 06:24:15.968413 10401 ts_manager.cc:194] Registered new tserver with Master: 87bf64a39a694b70ae53402ca3fc25cf (127.31.214.67:34411)
I20260928 06:24:15.968835 10401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:54225
I20260928 06:24:15.974525 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:24:15.982825 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.976076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.31.214.126:45391)
0928 06:24:15.976177 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:15.976179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:15.976321 (+ 142us) server_negotiation.cc:407] Connection header received
0928 06:24:15.976348 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:15.976349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:15.976360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:15.976415 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:24:15.977040 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.977580 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.978530 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.978804 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.978881 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.980282 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:24:15.980283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:24:15.980284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:24:15.980292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:24:15.981610 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.982089 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.982090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.982090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.982163 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:24:15.982346 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:24:15.982346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:24:15.982346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:24:15.982495 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:24:15.982568 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:15.982694 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:24:15.982758 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 06:24:15.984810 10401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:24:15.986819 10401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:24:15.993677 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.989347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:45979 (local address 127.0.0.1:39182)
0928 06:24:15.989862 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:15.989905 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:15.990028 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:15.990249 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:15.990252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:15.990254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:24:15.990332 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.990334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.991366 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.991367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.992338 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.992341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.993375 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.993376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.993457 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.993470 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:24:15.993540 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:24:15.993593 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":105,"threads_started":1}
I20260928 06:24:15.993727 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.989653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.31.214.66:45979)
0928 06:24:15.989869 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:15.989875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:15.990001 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:24:15.990130 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:15.990133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:15.990149 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:15.990244 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:24:15.990426 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.991270 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.992440 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.993163 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.993244 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.993308 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:15.993589 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 06:24:15.993637 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":98,"threads_started":1}
I20260928 06:24:15.994407 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.989347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:34411 (local address 127.0.0.1:40208)
0928 06:24:15.989863 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:15.989908 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:15.990059 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:15.990465 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:15.990466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:15.990468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:24:15.990572 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.990574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.992021 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.992022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.993077 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.993080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.994084 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.994105 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.994183 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.994188 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:24:15.994235 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:24:15.994281 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260928 06:24:15.994817 10660 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe14fdd190a4845b0dbbd404f59bc91 (DEFAULT_TABLE table=test-table [id=908824f52b5e4b7e9f1d91469c186ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:24:15.995205 10660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe14fdd190a4845b0dbbd404f59bc91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:24:15.995535 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.989356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46241 (local address 127.0.0.1:39016)
0928 06:24:15.990189 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:15.990199 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:15.990381 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:15.990613 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:15.990614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:15.990616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:24:15.990656 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.990658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.993199 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.993200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.994395 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:15.994398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.995300 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.995301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:15.995364 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.995369 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:24:15.995427 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:24:15.995468 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":112,"threads_started":1}
I20260928 06:24:15.995560 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.989653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.31.214.65:46241)
0928 06:24:15.989999 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:15.990003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:15.990443 (+ 440us) server_negotiation.cc:407] Connection header received
0928 06:24:15.990489 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:15.990492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:15.990508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:15.990604 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:24:15.990738 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.991905 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.994637 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.995196 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.995295 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.995355 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:15.995441 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 06:24:15.995481 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":91,"threads_started":1}
I20260928 06:24:15.996162 10526 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe14fdd190a4845b0dbbd404f59bc91 (DEFAULT_TABLE table=test-table [id=908824f52b5e4b7e9f1d91469c186ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:24:15.996457 10526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe14fdd190a4845b0dbbd404f59bc91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:24:15.998976 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:15.989611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.31.214.67:34411)
0928 06:24:15.989824 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:15.989830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:15.990024 (+ 194us) server_negotiation.cc:407] Connection header received
0928 06:24:15.990257 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:15.990260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:15.990281 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:15.990366 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:24:15.990660 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.991900 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.993201 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:15.993934 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:15.998681 (+ 4747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:15.998764 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:15.998797 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:24:15.998850 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":141,"threads_started":1}
I20260928 06:24:15.999754 10878 tablet_bootstrap.cc:492] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e: Bootstrap starting.
I20260928 06:24:16.000373 10878 tablet_bootstrap.cc:654] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e: Neither blocks nor log segments found. Creating new log.
I20260928 06:24:16.000910 10878 log.cc:824] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:24:16.001747 10878 tablet_bootstrap.cc:492] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e: No bootstrap required, opened a new log
I20260928 06:24:16.001839 10878 ts_tablet_manager.cc:1397] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:24:16.002362 10877 tablet_bootstrap.cc:492] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: Bootstrap starting.
I20260928 06:24:16.002840 10794 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe14fdd190a4845b0dbbd404f59bc91 (DEFAULT_TABLE table=test-table [id=908824f52b5e4b7e9f1d91469c186ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:24:16.003206 10877 tablet_bootstrap.cc:654] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: Neither blocks nor log segments found. Creating new log.
I20260928 06:24:16.003262 10794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe14fdd190a4845b0dbbd404f59bc91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:24:16.003886 10877 log.cc:824] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:24:16.004163 10878 raft_consensus.cc:348] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } }
I20260928 06:24:16.004335 10878 raft_consensus.cc:374] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:24:16.004374 10878 raft_consensus.cc:729] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5540ed40544e809d7688a236c3ed1e, State: Initialized, Role: FOLLOWER
I20260928 06:24:16.004539 10878 consensus_queue.cc:260] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } }
I20260928 06:24:16.004871 10878 ts_tablet_manager.cc:1428] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:24:16.005003 10592 heartbeater.cc:503] Master 127.31.214.126:45391 was elected leader, sending a full tablet report...
I20260928 06:24:16.006376 10877 tablet_bootstrap.cc:492] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: No bootstrap required, opened a new log
I20260928 06:24:16.006503 10877 ts_tablet_manager.cc:1397] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:24:16.006951 10882 tablet_bootstrap.cc:492] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: Bootstrap starting.
I20260928 06:24:16.007506 10882 tablet_bootstrap.cc:654] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: Neither blocks nor log segments found. Creating new log.
I20260928 06:24:16.007963 10882 log.cc:824] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:24:16.008252 10877 raft_consensus.cc:348] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } }
I20260928 06:24:16.008409 10877 raft_consensus.cc:374] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:24:16.008456 10877 raft_consensus.cc:729] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d415e9d76dd4f85bc49eb10a4b2911a, State: Initialized, Role: FOLLOWER
I20260928 06:24:16.008582 10877 consensus_queue.cc:260] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } }
I20260928 06:24:16.008708 10882 tablet_bootstrap.cc:492] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: No bootstrap required, opened a new log
I20260928 06:24:16.008786 10882 ts_tablet_manager.cc:1397] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:24:16.008929 10726 heartbeater.cc:503] Master 127.31.214.126:45391 was elected leader, sending a full tablet report...
I20260928 06:24:16.008910 10877 ts_tablet_manager.cc:1428] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:24:16.010677 10882 raft_consensus.cc:348] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } }
I20260928 06:24:16.010808 10882 raft_consensus.cc:374] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:24:16.010829 10882 raft_consensus.cc:729] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87bf64a39a694b70ae53402ca3fc25cf, State: Initialized, Role: FOLLOWER
I20260928 06:24:16.010943 10882 consensus_queue.cc:260] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } }
I20260928 06:24:16.011180 10882 ts_tablet_manager.cc:1428] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 06:24:16.011247 10860 heartbeater.cc:503] Master 127.31.214.126:45391 was elected leader, sending a full tablet report...
W20260928 06:24:16.042403 10727 tablet.cc:2369] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:24:16.103322 10884 raft_consensus.cc:482] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:24:16.103461 10884 raft_consensus.cc:504] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } }
I20260928 06:24:16.103904 10884 leader_election.cc:288] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87bf64a39a694b70ae53402ca3fc25cf (127.31.214.67:34411), 7b5540ed40544e809d7688a236c3ed1e (127.31.214.65:46241)
I20260928 06:24:16.107277 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:16.104155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:34411 (local address 127.31.214.66:35597)
0928 06:24:16.104251 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:16.104264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:16.104315 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:16.104521 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:16.104523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:16.104525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:24:16.104604 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:16.104606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.105401 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.105402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:16.106299 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:16.106301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.106952 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.106953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:16.107041 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:16.107045 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:24:16.107108 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:24:16.107157 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 06:24:16.107280 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:16.104286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:35597 (local address 127.31.214.67:34411)
0928 06:24:16.104334 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:16.104336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:16.104350 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:24:16.104384 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:16.104385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:16.104397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:16.104488 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:24:16.104694 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.105310 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.106386 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.106864 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.106923 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:16.106931 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:16.107142 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 06:24:16.107203 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:24:16.107487 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:16.104155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46241 (local address 127.31.214.66:42257)
0928 06:24:16.104450 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:16.104461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:24:16.104525 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:24:16.104798 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:24:16.104799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:24:16.104801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:24:16.104875 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:16.104877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.105583 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.105584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:16.106510 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:24:16.106512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.107214 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.107215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:24:16.107288 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:16.107293 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:24:16.107350 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:24:16.107391 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":95,"thread_start_us":137,"threads_started":1}
I20260928 06:24:16.107645 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:16.104262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:42257 (local address 127.31.214.65:46241)
0928 06:24:16.104345 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:16.104348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:16.104542 (+ 194us) server_negotiation.cc:407] Connection header received
0928 06:24:16.104618 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:16.104620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:16.104630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:16.104764 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:24:16.104965 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.105494 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.106606 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.107128 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.107193 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:16.107202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:16.107526 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 06:24:16.107576 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:24:16.107695 10814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe14fdd190a4845b0dbbd404f59bc91" candidate_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87bf64a39a694b70ae53402ca3fc25cf" is_pre_election: true
I20260928 06:24:16.107892 10814 raft_consensus.cc:2454] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d415e9d76dd4f85bc49eb10a4b2911a in term 0.
I20260928 06:24:16.107965 10546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe14fdd190a4845b0dbbd404f59bc91" candidate_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5540ed40544e809d7688a236c3ed1e" is_pre_election: true
I20260928 06:24:16.108150 10546 raft_consensus.cc:2454] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d415e9d76dd4f85bc49eb10a4b2911a in term 0.
I20260928 06:24:16.108177 10613 leader_election.cc:302] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d415e9d76dd4f85bc49eb10a4b2911a, 87bf64a39a694b70ae53402ca3fc25cf; no voters:
I20260928 06:24:16.108302 10884 raft_consensus.cc:2781] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:24:16.108357 10884 raft_consensus.cc:482] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:24:16.108410 10884 raft_consensus.cc:3037] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:24:16.109180 10884 raft_consensus.cc:504] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } }
I20260928 06:24:16.109349 10884 leader_election.cc:288] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [CANDIDATE]: Term 1 election: Requested vote from peers 87bf64a39a694b70ae53402ca3fc25cf (127.31.214.67:34411), 7b5540ed40544e809d7688a236c3ed1e (127.31.214.65:46241)
I20260928 06:24:16.109522 10814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe14fdd190a4845b0dbbd404f59bc91" candidate_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87bf64a39a694b70ae53402ca3fc25cf"
I20260928 06:24:16.109601 10814 raft_consensus.cc:3037] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:24:16.109532 10546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe14fdd190a4845b0dbbd404f59bc91" candidate_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5540ed40544e809d7688a236c3ed1e"
I20260928 06:24:16.109767 10546 raft_consensus.cc:3037] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:24:16.110426 10814 raft_consensus.cc:2454] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d415e9d76dd4f85bc49eb10a4b2911a in term 1.
I20260928 06:24:16.110601 10613 leader_election.cc:302] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d415e9d76dd4f85bc49eb10a4b2911a, 87bf64a39a694b70ae53402ca3fc25cf; no voters:
I20260928 06:24:16.110693 10546 raft_consensus.cc:2454] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d415e9d76dd4f85bc49eb10a4b2911a in term 1.
I20260928 06:24:16.110706 10884 raft_consensus.cc:2781] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:24:16.110908 10884 raft_consensus.cc:686] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 1 LEADER]: Becoming Leader. State: Replica: 6d415e9d76dd4f85bc49eb10a4b2911a, State: Running, Role: LEADER
I20260928 06:24:16.111016 10884 consensus_queue.cc:237] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } }
I20260928 06:24:16.111965 10400 catalog_manager.cc:5495] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d415e9d76dd4f85bc49eb10a4b2911a (127.31.214.66). New cstate: current_term: 1 leader_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d415e9d76dd4f85bc49eb10a4b2911a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 } health_report { overall_health: UNKNOWN } } }
W20260928 06:24:16.118201 10593 tablet.cc:2369] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:24:16.145355 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:16.142921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.31.214.66:45979)
0928 06:24:16.142998 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:24:16.143000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:24:16.143008 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:24:16.143044 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:24:16.143045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:24:16.143062 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:24:16.143149 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:24:16.143292 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.143817 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.144585 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:24:16.144807 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:24:16.144865 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:24:16.145010 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:24:16.145110 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:24:16.145179 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:24:16.145214 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:24:16.145232 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:24:16.147456 10814 raft_consensus.cc:1261] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [term 1 FOLLOWER]: Refusing update from remote peer 6d415e9d76dd4f85bc49eb10a4b2911a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:24:16.147473 10546 raft_consensus.cc:1261] T 6fe14fdd190a4845b0dbbd404f59bc91 P 7b5540ed40544e809d7688a236c3ed1e [term 1 FOLLOWER]: Refusing update from remote peer 6d415e9d76dd4f85bc49eb10a4b2911a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:24:16.147882 10884 consensus_queue.cc:1035] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87bf64a39a694b70ae53402ca3fc25cf" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:24:16.148066 10887 consensus_queue.cc:1035] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5540ed40544e809d7688a236c3ed1e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:24:16.150352 10890 mvcc.cc:205] Tried to move back new op lower bound from 7334201983577169920 to 7334201983431606272. Current Snapshot: MvccSnapshot[applied={T|T < 7334201983577169920}]
I20260928 06:24:16.151648 10891 mvcc.cc:205] Tried to move back new op lower bound from 7334201983577169920 to 7334201983431606272. Current Snapshot: MvccSnapshot[applied={T|T < 7334201983577169920}]
I20260928 06:24:16.151983 10893 mvcc.cc:205] Tried to move back new op lower bound from 7334201983577169920 to 7334201983431606272. Current Snapshot: MvccSnapshot[applied={T|T < 7334201983577169920}]
I20260928 06:24:16.152781 10401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:24:16.152886 10401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:24:16.153750 10401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f62293a433f4490951a7e56d9a47cc3: Sending DeleteTablet for 3 replicas of tablet 6fe14fdd190a4845b0dbbd404f59bc91
I20260928 06:24:16.154173 10794 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe14fdd190a4845b0dbbd404f59bc91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:24:16 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:40208
I20260928 06:24:16.154170 10660 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe14fdd190a4845b0dbbd404f59bc91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:24:16 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:39182
I20260928 06:24:16.154201 10526 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe14fdd190a4845b0dbbd404f59bc91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:24:16 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:39016
I20260928 06:24:16.154464 10902 tablet_replica.cc:330] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: stopping tablet replica
I20260928 06:24:16.154572 10902 raft_consensus.cc:2229] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:24:16.154680 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 10463
I20260928 06:24:16.154699 10902 raft_consensus.cc:2258] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:24:16.154873 10900 tablet_replica.cc:330] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: stopping tablet replica
I20260928 06:24:16.154968 10900 raft_consensus.cc:2229] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:24:16.155057 10900 raft_consensus.cc:2258] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:24:16.155717 10900 ts_tablet_manager.cc:1905] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:24:16.157744 10900 ts_tablet_manager.cc:1918] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:24:16.157829 10900 log.cc:1196] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/6fe14fdd190a4845b0dbbd404f59bc91
I20260928 06:24:16.158169 10900 ts_tablet_manager.cc:1939] T 6fe14fdd190a4845b0dbbd404f59bc91 P 6d415e9d76dd4f85bc49eb10a4b2911a: Deleting consensus metadata
I20260928 06:24:16.158378 10902 ts_tablet_manager.cc:1905] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:24:16.158705 10387 catalog_manager.cc:4841] TS 6d415e9d76dd4f85bc49eb10a4b2911a (127.31.214.66:45979): tablet 6fe14fdd190a4845b0dbbd404f59bc91 (table test-table [id=908824f52b5e4b7e9f1d91469c186ea6]) successfully deleted
I20260928 06:24:16.160020 10902 ts_tablet_manager.cc:1918] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:24:16.160104 10902 log.cc:1196] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/6fe14fdd190a4845b0dbbd404f59bc91
I20260928 06:24:16.160419 10902 ts_tablet_manager.cc:1939] T 6fe14fdd190a4845b0dbbd404f59bc91 P 87bf64a39a694b70ae53402ca3fc25cf: Deleting consensus metadata
I20260928 06:24:16.160928 10385 catalog_manager.cc:4841] TS 87bf64a39a694b70ae53402ca3fc25cf (127.31.214.67:34411): tablet 6fe14fdd190a4845b0dbbd404f59bc91 (table test-table [id=908824f52b5e4b7e9f1d91469c186ea6]) successfully deleted
W20260928 06:24:16.162128 10388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:46241 (error 108)
I20260928 06:24:16.162488 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 10597
I20260928 06:24:16.163085 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:24:16.162558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46241 (local address 127.0.0.1:39020)
0928 06:24:16.163010 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 06:24:16.163049 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
W20260928 06:24:16.165114 10388 catalog_manager.cc:4584] TS 7b5540ed40544e809d7688a236c3ed1e (127.31.214.65:46241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fe14fdd190a4845b0dbbd404f59bc91: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46241: connect: Connection refused (error 111)
I20260928 06:24:16.168686 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 10731
I20260928 06:24:16.173504 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 10369
2026-09-28T06:24:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2136 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:24:16 dist-test-slave-q03f krb5kdc[10906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:24:16 dist-test-slave-q03f krb5kdc[10906](info): set up 2 sockets
Sep 28 06:24:16 dist-test-slave-q03f krb5kdc[10906](info): commencing operation
krb5kdc: starting...
W20260928 06:24:17.222447 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:24:17 dist-test-slave-q03f krb5kdc[10906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:24:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:24:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.214.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.214.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/rangeradmin_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/rangeradmin_127.31.214.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.214.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.214.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/rangerlookup_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/rangerlookup_127.31.214.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.214.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.214.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/HTTP_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/HTTP_127.31.214.84@KRBTEST.COM.keytab.
I20260928 06:24:17.397784 32601 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDYGmsK/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/postgres -l logfile start
2026-09-28 06:24:20.281 UTC [10936] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:24:20.281 UTC [10936] LOG: listening on IPv4 address "127.31.214.84", port 55695
2026-09-28 06:24:20.287 UTC [10936] LOG: listening on Unix socket "/tmp/.s.PGSQL.55695"
2026-09-28 06:24:20.326 UTC [10939] LOG: database system was shut down at 2026-09-28 06:24:18 UTC
2026-09-28 06:24:20.330 UTC [10936] LOG: database system is ready to accept connections
I20260928 06:24:21.266563 32601 mini_postgres.cc:96] Postgres bound to 55695
2026-09-28 06:24:21.277 UTC [10947] FATAL: database "slave" does not exist
127.31.214.84:55695 - accepting connections
I20260928 06:24:21.278079 32601 mini_ranger.cc:161] Starting Ranger...
I20260928 06:24:21.296325 32601 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:24:21.859367 32601 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:24:21.859462 32601 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-admin
2026-09-28 06:24:22,175 [I] DB FLAVOR :POSTGRES
2026-09-28 06:24:22,175 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:24:22,175 [I] Checking connection..
2026-09-28 06:24:22,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:24:22,459 [I] Checking connection passed.
2026-09-28 06:24:22,459 [I] --------- Verifying version history table ---------
2026-09-28 06:24:22,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:24:22,722 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:24:22,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:24:23,001 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:24:23,001 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:24:23,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:24:23.233 UTC [11011] WARNING: there is no transaction in progress
2026-09-28 06:24:23,240 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:24:23,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:24:23,517 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:24:23,517 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:24:23,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:24:23,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:24:24,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q03f.c.gcp-upstream.internal','N') ;"
2026-09-28 06:24:24,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:24:24,522 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:24:24,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:24:24,793 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:24:24,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:24:25,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:24:25,301 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:24:25,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:24:26.403 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.421 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.440 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.458 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.476 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.496 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.560 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.570 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.586 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.599 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.611 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.627 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.638 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.645 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.653 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:26.661 UTC [11119] WARNING: there is no transaction in progress
2026-09-28 06:24:27,458 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:24:27,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:24:27,699 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:24:27,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:24:27,980 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:24:27,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:24:28,232 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:24:28,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:24:28,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:24:28,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q03f.c.gcp-upstream.internal';"
2026-09-28 06:24:28,949 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:24:28,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:24:29,186 [I] DB_PATCHES have already been applied
I20260928 06:24:29.192662 32601 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDYGmsK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:24:29.192739 32601 mini_ranger.cc:193] Using host: 127.31.214.84
I20260928 06:24:29.196362 32601 mini_ranger.cc:227] Ranger admin URL: http://127.31.214.84:43605
Sep 28, 2026 6:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:24:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:24:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:24:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:24:30.308920 32601 mini_ranger.cc:160] Time spent starting Ranger: real 9.031s user 0.002s sys 0.008s
Sep 28, 2026 6:24:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:24:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 6:24:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:24:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:24:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:24:54.151686 32601 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.214.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.214.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/rangerkms_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/rangerkms_127.31.214.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.214.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.214.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.214.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/HTTP_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.214.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/HTTP_127.31.214.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:24:54 dist-test-slave-q03f krb5kdc[10906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576694, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:24:54.252496 32601 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:24:54.264014 32601 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:24:55.334564 32601 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:24:55.334666 32601 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-kms
I20260928 06:24:55.334686 32601 mini_ranger_kms.cc:227] Using postgres at 127.31.214.84:55695
2026-09-28 06:24:56,110 [I] DB FLAVOR :POSTGRES
2026-09-28 06:24:56,110 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:24:56,110 [I] Checking connection
2026-09-28 06:24:56,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:24:56,410 [I] connection success
2026-09-28 06:24:56,410 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:24:56,410 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:24:56,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:24:56,662 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:24:56,662 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:24:56,662 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:24:56,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:55695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:24:56,946 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:24:57.504622 32601 mini_ranger_kms.cc:310] Created kms service
I20260928 06:24:57.728777 32601 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:24:57.774118 32601 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:24:57 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576694, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:24:58.209411 32601 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:24:58.209539 32601 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:24:58.209568 32601 mini_ranger_kms.cc:240] Using host: 127.31.214.84
I20260928 06:24:58.213567 32601 mini_ranger_kms.cc:276] Ranger KMS PID: 11300
I20260928 06:24:58.213685 32601 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.214.84:57305
Sep 28, 2026 6:24:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:24:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:24:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:24:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:24:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:24:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:25:00.584414 32601 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.332s user 0.017s sys 0.273s
I20260928 06:25:00.584540 32601 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:25:00.584638 32601 mini_ranger_kms.cc:401] 127.31.214.84:57305/kms/v1/keys
Sep 28 06:25:00 dist-test-slave-q03f krb5kdc[10906](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790576700, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.214.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:25:01 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576700, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.214.84@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
Sep 28 06:25:01 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576700, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.214.84@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
Sep 28 06:25:04 dist-test-slave-q03f krb5kdc[10906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:25:04.722541 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35093
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:35093
--ranger_config_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:57305/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:25:04.855815 11350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:04.856135 11350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:04.856180 11350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:04.858328 11350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:25:04.858421 11350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:04.858449 11350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:04.858470 11350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:25:04.858500 11350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:25:04.861423 11350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.214.84:57305/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:35093
--ranger_config_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:04.861742 11350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:04.862092 11350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:04.868769 11356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:04.868878 11350 server_base.cc:1034] running on GCE node
W20260928 06:25:04.868957 11355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:04.868785 11358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:04.869364 11350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:04.869745 11350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:04.870939 11350 hybrid_clock.cc:648] HybridClock initialized: now 1790576704870883 us; error 74 us; skew 500 ppm
Sep 28 06:25:04 dist-test-slave-q03f krb5kdc[10906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576704, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:04.879906 11350 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:25:04.880647 11350 webserver.cc:460] Webserver started at http://127.31.214.126:37081/ using document root <none> and password file <none>
I20260928 06:25:04.881008 11350 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:04.881071 11350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:25:04.881213 11350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:25:04 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576704, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:04.955229 11350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "046373ea17584c739b0be901555e7091"
format_stamp: "Formatted at 2026-09-28 06:25:04 on dist-test-slave-q03f"
server_key: "625845d37105dc2446c17bfae44d0a7c"
server_key_iv: "a14a9046f8abbf9fdaa2dad8062d02f7"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:04.955744 11350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "046373ea17584c739b0be901555e7091"
format_stamp: "Formatted at 2026-09-28 06:25:04 on dist-test-slave-q03f"
server_key: "625845d37105dc2446c17bfae44d0a7c"
server_key_iv: "a14a9046f8abbf9fdaa2dad8062d02f7"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:04.957790 11350 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 06:25:04.975776 11365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:04.976023 11350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:25:04.976150 11350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "046373ea17584c739b0be901555e7091"
format_stamp: "Formatted at 2026-09-28 06:25:04 on dist-test-slave-q03f"
server_key: "625845d37105dc2446c17bfae44d0a7c"
server_key_iv: "a14a9046f8abbf9fdaa2dad8062d02f7"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:04.976281 11350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:04.994189 11350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:04.996613 11350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:04.996807 11350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:05.002585 11350 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:35093
I20260928 06:25:05.002606 11417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:35093 every 8 connection(s)
I20260928 06:25:05.003041 11350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:25:05.003798 11418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:25:05.006346 11418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091: Bootstrap starting.
I20260928 06:25:05.006948 11418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091: Neither blocks nor log segments found. Creating new log.
I20260928 06:25:05.007256 11418 log.cc:824] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091: Log is configured to *not* fsync() on all Append() calls
I20260928 06:25:05.007967 11418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091: No bootstrap required, opened a new log
I20260928 06:25:05.010186 11418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046373ea17584c739b0be901555e7091" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35093 } }
I20260928 06:25:05.010365 11418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:25:05.010414 11418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 046373ea17584c739b0be901555e7091, State: Initialized, Role: FOLLOWER
I20260928 06:25:05.010607 11418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046373ea17584c739b0be901555e7091" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35093 } }
I20260928 06:25:05.010710 11418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:25:05.010752 11418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:25:05.010800 11418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:25:05.011066 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 11350
I20260928 06:25:05.011178 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:25:05.011413 32601 external_mini_cluster.cc:1428] Setting key 48726ff95b2ff60e6ceb51d0ce672056
I20260928 06:25:05.011620 11418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046373ea17584c739b0be901555e7091" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35093 } }
I20260928 06:25:05.011730 11418 leader_election.cc:302] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 046373ea17584c739b0be901555e7091; no voters:
I20260928 06:25:05.011899 11418 leader_election.cc:288] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:25:05.012171 11418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:25:05.012485 11423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:25:05.012637 11423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [term 1 LEADER]: Becoming Leader. State: Replica: 046373ea17584c739b0be901555e7091, State: Running, Role: LEADER
I20260928 06:25:05.012737 11423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046373ea17584c739b0be901555e7091" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35093 } }
I20260928 06:25:05.013100 11423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 046373ea17584c739b0be901555e7091. Latest consensus state: current_term: 1 leader_uuid: "046373ea17584c739b0be901555e7091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046373ea17584c739b0be901555e7091" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35093 } } }
I20260928 06:25:05.013195 11423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [sys.catalog]: This master's current role is: LEADER
I20260928 06:25:05.013303 11425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "046373ea17584c739b0be901555e7091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046373ea17584c739b0be901555e7091" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35093 } } }
I20260928 06:25:05.013417 11434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:25:05.013437 11425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091 [sys.catalog]: This master's current role is: LEADER
I20260928 06:25:05.013698 11418 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:25:05.016969 11434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:25:05.020251 11434 catalog_manager.cc:1347] Generated new cluster ID: 458d36104bb342f78306c887161f3286
I20260928 06:25:05.020301 11434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:25:05 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:05.036725 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:05.012077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.31.214.126:35093)
0928 06:25:05.014043 (+ 1966us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:05.014053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:05.014089 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:25:05.016163 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:05.016256 (+ 93us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:05.018164 (+ 1908us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:05.019269 (+ 1105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:05.019523 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:05.021376 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:05.023011 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:05.024151 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:05.029097 (+ 4946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:05.029371 (+ 274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:05.029376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:05.029376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:05.029390 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:05.033142 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:05.033966 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:05.033971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:05.033971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:05.034101 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:05.034504 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:05.034505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:05.034505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:05.035138 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:05.036084 (+ 946us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:05.036282 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:25:05.036442 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1911,"thread_start_us":101,"threads_started":1}
I20260928 06:25:05.058297 11434 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:25:05.059037 11434 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:25:05.066583 11434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091: Generated new TSK 0
I20260928 06:25:05.066782 11434 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:25:05.942986 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:57305/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35093
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:25:06.090394 11463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:06.090701 11463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:06.090737 11463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:06.092451 11463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:06.092533 11463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:06.092572 11463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:25:06.094831 11463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.214.84:57305/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:06.095158 11463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:06.095453 11463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:06.102850 11468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:06.102892 11469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:06.103395 11463 server_base.cc:1034] running on GCE node
W20260928 06:25:06.106089 11471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:06.106419 11463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:06.107031 11463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:06.108253 11463 hybrid_clock.cc:648] HybridClock initialized: now 1790576706108217 us; error 52 us; skew 500 ppm
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:06.113445 11463 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:25:06.114341 11463 webserver.cc:460] Webserver started at http://127.31.214.65:46631/ using document root <none> and password file <none>
I20260928 06:25:06.115062 11463 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:06.115139 11463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:25:06.115269 11463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:06.142468 11463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "36a02c2f09b740d49cf62a22dc5020ee"
format_stamp: "Formatted at 2026-09-28 06:25:06 on dist-test-slave-q03f"
server_key: "299f7353cafe60e82d37695fd77baeac"
server_key_iv: "4f51db38476fe13634f4edcce59b3681"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:06.143097 11463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "36a02c2f09b740d49cf62a22dc5020ee"
format_stamp: "Formatted at 2026-09-28 06:25:06 on dist-test-slave-q03f"
server_key: "299f7353cafe60e82d37695fd77baeac"
server_key_iv: "4f51db38476fe13634f4edcce59b3681"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:06.145299 11463 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:25:06.161597 11478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:06.161829 11463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:25:06.162043 11463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "36a02c2f09b740d49cf62a22dc5020ee"
format_stamp: "Formatted at 2026-09-28 06:25:06 on dist-test-slave-q03f"
server_key: "299f7353cafe60e82d37695fd77baeac"
server_key_iv: "4f51db38476fe13634f4edcce59b3681"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:06.162150 11463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:06.192602 11463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:06.197484 11463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:06.197671 11463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:06.198197 11463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:25:06.198689 11463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:25:06.198737 11463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:06.198873 11463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:25:06.198900 11463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:06.208828 11463 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:35867
I20260928 06:25:06.209612 11463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:25:06.211934 11593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:35867 every 8 connection(s)
I20260928 06:25:06.212450 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 11463
I20260928 06:25:06.212720 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:25:06.213224 32601 external_mini_cluster.cc:1428] Setting key 03b55979e0d44ac2071d4375fd518486
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:06.225669 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.212811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:48855 (local address 127.31.214.126:35093)
0928 06:25:06.212944 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.212947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.213266 (+ 319us) server_negotiation.cc:407] Connection header received
0928 06:25:06.214190 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.214191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.214206 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.214249 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:06.215174 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.215972 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.217576 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.217876 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.217932 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.221401 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:06.221403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:06.221403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:06.221412 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:06.223538 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:06.224166 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:06.224168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:06.224168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:06.224243 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:06.224617 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:06.224619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:06.224619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:06.224791 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:06.225029 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.225459 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.225555 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260928 06:25:06.226758 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.212753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35093 (local address 127.31.214.65:48855)
0928 06:25:06.213113 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:06.213166 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:06.214063 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:06.214448 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:06.214453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:06.214763 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:25:06.215058 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.215066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.216094 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.216095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.217460 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.217464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.218047 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.218048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.218186 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.218823 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:25:06.218842 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:25:06.221272 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:25:06.223648 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:06.223657 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:06.223660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:06.224048 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:06.224353 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:06.224353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:06.224354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:06.224525 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:06.225080 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:25:06.225087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:25:06.225276 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 06:25:06.226269 (+ 993us) client_negotiation.cc:240] Negotiation successful
0928 06:25:06.226476 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":83,"threads_started":1}
I20260928 06:25:06.227572 11594 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35093
I20260928 06:25:06.227694 11594 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:06.227916 11594 heartbeater.cc:511] Master 127.31.214.126:35093 requested a full tablet report, sending...
I20260928 06:25:06.228570 11382 ts_manager.cc:194] Registered new tserver with Master: 36a02c2f09b740d49cf62a22dc5020ee (127.31.214.65:35867)
I20260928 06:25:06.229503 11382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:48855
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:25:06.258059 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:57305/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35093
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:25:06.391270 11600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:06.391572 11600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:06.391618 11600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:06.393431 11600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:06.393502 11600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:06.393553 11600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:25:06.395885 11600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.214.84:57305/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:06.396203 11600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:06.396514 11600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:06.406889 11607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:06.407639 11608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:06.408645 11600 server_base.cc:1034] running on GCE node
W20260928 06:25:06.408917 11610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:06.409252 11600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:06.409924 11600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:06.411147 11600 hybrid_clock.cc:648] HybridClock initialized: now 1790576706411123 us; error 41 us; skew 500 ppm
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:06.415139 11600 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:25:06.415872 11600 webserver.cc:460] Webserver started at http://127.31.214.66:39191/ using document root <none> and password file <none>
I20260928 06:25:06.416183 11600 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:06.416302 11600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:25:06.416486 11600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:06.434818 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "c9ea9713c9084934ad55812cfa50fbcc"
format_stamp: "Formatted at 2026-09-28 06:25:06 on dist-test-slave-q03f"
server_key: "9a5c96ff84fe6a7fe3ab03fab618cc43"
server_key_iv: "9a59489bb25bceda9ac58ef260cdeebd"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:06.435333 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "c9ea9713c9084934ad55812cfa50fbcc"
format_stamp: "Formatted at 2026-09-28 06:25:06 on dist-test-slave-q03f"
server_key: "9a5c96ff84fe6a7fe3ab03fab618cc43"
server_key_iv: "9a59489bb25bceda9ac58ef260cdeebd"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:06.438668 11600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 06:25:06.458691 11617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:06.459052 11600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260928 06:25:06.459167 11600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "c9ea9713c9084934ad55812cfa50fbcc"
format_stamp: "Formatted at 2026-09-28 06:25:06 on dist-test-slave-q03f"
server_key: "9a5c96ff84fe6a7fe3ab03fab618cc43"
server_key_iv: "9a59489bb25bceda9ac58ef260cdeebd"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:06.459408 11600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:06.482653 11600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:06.486065 11600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:06.486253 11600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:06.486510 11600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:25:06.487046 11600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:25:06.487098 11600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:06.487236 11600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:25:06.487265 11600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:06.501070 11600 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:41583
I20260928 06:25:06.501333 11730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:41583 every 8 connection(s)
I20260928 06:25:06.501579 11600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:25:06.509392 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 11600
I20260928 06:25:06.509524 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:25:06.509745 32601 external_mini_cluster.cc:1428] Setting key b076bcd5aed44055c98129d09c32e669
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:06.516635 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.502823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:40581 (local address 127.31.214.126:35093)
0928 06:25:06.502871 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.502874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.502886 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:25:06.503619 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.503621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.503712 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.503782 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:06.504640 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.505478 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.507246 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.507568 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.507770 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.512508 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:06.512510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:06.512510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:06.512519 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:06.514466 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:06.515184 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:06.515185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:06.515186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:06.515257 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:06.515639 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:06.515640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:06.515640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:06.515793 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:06.515872 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.516285 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.516385 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:25:06.516709 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.502533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35093 (local address 127.31.214.66:40581)
0928 06:25:06.502753 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:06.502803 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:06.503490 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:06.503890 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:06.503895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:06.504181 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:25:06.504476 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.504483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.505622 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.505623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.507137 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.507140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.507791 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.507792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.507954 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.508642 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:25:06.508653 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:25:06.510684 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:25:06.514613 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:06.514623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:06.514626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:06.515008 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:06.515341 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:06.515342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:06.515343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:06.515531 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:06.515885 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:25:06.515892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:25:06.516114 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 06:25:06.516245 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 06:25:06.516449 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260928 06:25:06.517570 11731 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35093
I20260928 06:25:06.517755 11731 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:06.518085 11731 heartbeater.cc:511] Master 127.31.214.126:35093 requested a full tablet report, sending...
I20260928 06:25:06.518651 11382 ts_manager.cc:194] Registered new tserver with Master: c9ea9713c9084934ad55812cfa50fbcc (127.31.214.66:41583)
I20260928 06:25:06.519064 11382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:40581
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:25:06.549129 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:57305/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35093
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:25:06.668497 11737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:06.668874 11737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:06.668952 11737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:06.670828 11737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:06.670943 11737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:06.671017 11737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:25:06.673406 11737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.214.84:57305/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:06.673853 11737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:06.674213 11737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:06.681293 11745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:06.681288 11742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:06.681640 11743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:06.681757 11737 server_base.cc:1034] running on GCE node
I20260928 06:25:06.682114 11737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:06.682464 11737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:06.683655 11737 hybrid_clock.cc:648] HybridClock initialized: now 1790576706683632 us; error 37 us; skew 500 ppm
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:06.687386 11737 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:25:06.688084 11737 webserver.cc:460] Webserver started at http://127.31.214.67:43985/ using document root <none> and password file <none>
I20260928 06:25:06.688455 11737 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:06.688539 11737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:25:06.688774 11737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:06.720489 11737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "719c0cfceaa94021985d6357ec1f1633"
format_stamp: "Formatted at 2026-09-28 06:25:06 on dist-test-slave-q03f"
server_key: "092f7369c09332d0ec06765e5a06ed42"
server_key_iv: "5010a8b9ed63f1ebd4e063c272fdd746"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:06.721390 11737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "719c0cfceaa94021985d6357ec1f1633"
format_stamp: "Formatted at 2026-09-28 06:25:06 on dist-test-slave-q03f"
server_key: "092f7369c09332d0ec06765e5a06ed42"
server_key_iv: "5010a8b9ed63f1ebd4e063c272fdd746"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:06.726130 11737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 06:25:06.750512 11752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:06.750779 11737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:25:06.750890 11737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "719c0cfceaa94021985d6357ec1f1633"
format_stamp: "Formatted at 2026-09-28 06:25:06 on dist-test-slave-q03f"
server_key: "092f7369c09332d0ec06765e5a06ed42"
server_key_iv: "5010a8b9ed63f1ebd4e063c272fdd746"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:06.750989 11737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:06.776762 11737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:06.779399 11737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:06.779600 11737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:06.779855 11737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:25:06.780261 11737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:25:06.780334 11737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:06.780383 11737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:25:06.780418 11737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:06.787034 11737 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:44289
I20260928 06:25:06.787560 11737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:25:06.795362 11865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:44289 every 8 connection(s)
I20260928 06:25:06.797480 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 11737
I20260928 06:25:06.797596 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:25:06.797816 32601 external_mini_cluster.cc:1428] Setting key 23055943eab918fac62c5c74702cc768
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:06.808858 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.794858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35093 (local address 127.31.214.67:38535)
0928 06:25:06.796353 (+ 1495us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:06.796411 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:06.797144 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:06.798367 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:06.798372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:06.798639 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:25:06.798950 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.798957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.800054 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.800055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.801366 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.801369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.802116 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.802116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.802263 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.802935 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:25:06.802946 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:25:06.804744 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:25:06.806828 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:06.806836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:06.806838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:06.807150 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:06.807439 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:06.807440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:06.807441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:06.807600 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:06.808083 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:25:06.808091 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:25:06.808273 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 06:25:06.808408 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 06:25:06.808588 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1440,"thread_start_us":109,"threads_started":1}
I20260928 06:25:06.808876 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.795210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:38535 (local address 127.31.214.126:35093)
0928 06:25:06.795258 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.795261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.796512 (+ 1251us) server_negotiation.cc:407] Connection header received
0928 06:25:06.798097 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.798098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.798113 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.798149 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:06.799063 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.799927 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.801473 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.801803 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.801848 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.804878 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:06.804880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:06.804880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:06.804888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:06.806691 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:06.807258 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:06.807260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:06.807260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:06.807337 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:06.807693 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:06.807695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:06.807695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:06.807984 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:06.808071 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.808557 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.808660 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:25:06.809715 11866 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35093
I20260928 06:25:06.810009 11866 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:06.810367 11866 heartbeater.cc:511] Master 127.31.214.126:35093 requested a full tablet report, sending...
I20260928 06:25:06.810993 11381 ts_manager.cc:194] Registered new tserver with Master: 719c0cfceaa94021985d6357ec1f1633 (127.31.214.67:44289)
I20260928 06:25:06.811486 11381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:38535
I20260928 06:25:06.821672 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:25:06.832491 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.823661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.31.214.126:35093)
0928 06:25:06.823715 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.823717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.824142 (+ 425us) server_negotiation.cc:407] Connection header received
0928 06:25:06.824231 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.824232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.824246 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.824280 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:06.825197 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.826006 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.827153 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.827462 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.827511 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.829208 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:06.829209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:06.829210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:06.829218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:06.831084 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:06.831525 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:06.831526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:06.831527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:06.831589 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:06.831810 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:06.831811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:06.831812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:06.831947 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:06.832022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.832299 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.832386 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:25:06.836227 11381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:25:06.839505 11381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:25:06.852209 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.842885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:41583 (local address 127.0.0.1:44218)
0928 06:25:06.845831 (+ 2946us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:06.845894 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:06.846072 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:06.846472 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:06.846475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:06.846478 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:06.846573 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.846575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.848570 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.848571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.850281 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.850285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.851847 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.851848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.851945 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.851963 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 06:25:06.852038 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:25:06.852101 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2897,"thread_start_us":86,"threads_started":1}
I20260928 06:25:06.852392 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.843531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.31.214.66:41583)
0928 06:25:06.843903 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.843910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.846016 (+ 2106us) server_negotiation.cc:407] Connection header received
0928 06:25:06.846305 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.846309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.846330 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.846440 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:06.846650 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.848462 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.850433 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.851739 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.851835 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.852082 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.852118 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.852164 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":244,"threads_started":1}
I20260928 06:25:06.853754 11665 tablet_service.cc:1467] Processing CreateTablet for tablet 90f7985271c0489299726dd3e536171c (DEFAULT_TABLE table=test-table [id=eacb5eec780c4da2a1887421b6de9067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:25:06.854245 11665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f7985271c0489299726dd3e536171c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:25:06.859565 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.842952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:35867 (local address 127.0.0.1:38830)
0928 06:25:06.846206 (+ 3254us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:06.846216 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:06.846258 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:06.847735 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:06.847737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:06.847739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:06.847791 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.847793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.849350 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.849351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.850691 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.850694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.859231 (+ 8537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.859232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.859316 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.859322 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:25:06.859400 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:25:06.859455 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3235,"thread_start_us":85,"threads_started":1}
I20260928 06:25:06.860183 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.843353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.31.214.65:35867)
0928 06:25:06.847472 (+ 4119us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.847478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.847492 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:25:06.847530 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.847532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.847553 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.847647 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:06.848064 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.849239 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.858035 (+ 8796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.859021 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.859106 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.859180 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.860044 (+ 864us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.860096 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4073,"thread_start_us":106,"threads_started":1}
I20260928 06:25:06.861110 11528 tablet_service.cc:1467] Processing CreateTablet for tablet 90f7985271c0489299726dd3e536171c (DEFAULT_TABLE table=test-table [id=eacb5eec780c4da2a1887421b6de9067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:25:06.861537 11528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f7985271c0489299726dd3e536171c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:25:06.863708 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.843468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.31.214.67:44289)
0928 06:25:06.858227 (+ 14759us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.858233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.858246 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:25:06.858290 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.858292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.858312 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.858530 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:06.859341 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.860532 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.861933 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.863207 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.863259 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.863343 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.863571 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.863617 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14713,"thread_start_us":95,"threads_started":1}
I20260928 06:25:06.863989 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.843218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:44289 (local address 127.0.0.1:37858)
0928 06:25:06.852941 (+ 9723us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:06.852953 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:06.852990 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:06.859144 (+ 6154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:06.859146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:06.859148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:06.859194 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.859196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.860646 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.860647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.861796 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.861800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.863409 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.863410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.863473 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.863477 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:25:06.863830 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 06:25:06.863884 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9708,"thread_start_us":90,"threads_started":1}
I20260928 06:25:06.864698 11800 tablet_service.cc:1467] Processing CreateTablet for tablet 90f7985271c0489299726dd3e536171c (DEFAULT_TABLE table=test-table [id=eacb5eec780c4da2a1887421b6de9067]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:25:06.865039 11800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f7985271c0489299726dd3e536171c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:25:06.869164 11887 tablet_bootstrap.cc:492] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: Bootstrap starting.
I20260928 06:25:06.869484 11885 tablet_bootstrap.cc:492] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc: Bootstrap starting.
I20260928 06:25:06.870121 11886 tablet_bootstrap.cc:492] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee: Bootstrap starting.
I20260928 06:25:06.871089 11886 tablet_bootstrap.cc:654] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee: Neither blocks nor log segments found. Creating new log.
I20260928 06:25:06.871516 11886 log.cc:824] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee: Log is configured to *not* fsync() on all Append() calls
I20260928 06:25:06.871562 11887 tablet_bootstrap.cc:654] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: Neither blocks nor log segments found. Creating new log.
I20260928 06:25:06.872316 11887 log.cc:824] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: Log is configured to *not* fsync() on all Append() calls
I20260928 06:25:06.872561 11885 tablet_bootstrap.cc:654] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc: Neither blocks nor log segments found. Creating new log.
I20260928 06:25:06.872895 11885 log.cc:824] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:25:06.873850 11886 tablet_bootstrap.cc:492] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee: No bootstrap required, opened a new log
I20260928 06:25:06.873939 11886 ts_tablet_manager.cc:1397] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:25:06.874378 11885 tablet_bootstrap.cc:492] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc: No bootstrap required, opened a new log
I20260928 06:25:06.875296 11885 ts_tablet_manager.cc:1397] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 06:25:06.876940 11886 raft_consensus.cc:348] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } }
I20260928 06:25:06.877125 11886 raft_consensus.cc:374] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:25:06.877163 11886 raft_consensus.cc:729] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a02c2f09b740d49cf62a22dc5020ee, State: Initialized, Role: FOLLOWER
I20260928 06:25:06.877398 11886 consensus_queue.cc:260] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } }
I20260928 06:25:06.877734 11886 ts_tablet_manager.cc:1428] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:25:06.877866 11594 heartbeater.cc:503] Master 127.31.214.126:35093 was elected leader, sending a full tablet report...
I20260928 06:25:06.880113 11887 tablet_bootstrap.cc:492] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: No bootstrap required, opened a new log
I20260928 06:25:06.880272 11887 ts_tablet_manager.cc:1397] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20260928 06:25:06.882777 11885 raft_consensus.cc:348] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } }
I20260928 06:25:06.882972 11885 raft_consensus.cc:374] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:25:06.883011 11885 raft_consensus.cc:729] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ea9713c9084934ad55812cfa50fbcc, State: Initialized, Role: FOLLOWER
I20260928 06:25:06.883168 11885 consensus_queue.cc:260] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } }
I20260928 06:25:06.883481 11885 ts_tablet_manager.cc:1428] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260928 06:25:06.883606 11731 heartbeater.cc:503] Master 127.31.214.126:35093 was elected leader, sending a full tablet report...
I20260928 06:25:06.885020 11887 raft_consensus.cc:348] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } }
I20260928 06:25:06.885676 11887 raft_consensus.cc:374] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:25:06.885850 11887 raft_consensus.cc:729] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719c0cfceaa94021985d6357ec1f1633, State: Initialized, Role: FOLLOWER
I20260928 06:25:06.886494 11887 consensus_queue.cc:260] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } }
I20260928 06:25:06.887442 11887 ts_tablet_manager.cc:1428] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260928 06:25:06.887632 11866 heartbeater.cc:503] Master 127.31.214.126:35093 was elected leader, sending a full tablet report...
I20260928 06:25:06.912533 11892 raft_consensus.cc:482] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:25:06.912693 11892 raft_consensus.cc:504] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } }
I20260928 06:25:06.913153 11892 leader_election.cc:288] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36a02c2f09b740d49cf62a22dc5020ee (127.31.214.65:35867), 719c0cfceaa94021985d6357ec1f1633 (127.31.214.67:44289)
I20260928 06:25:06.917841 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.913352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:35867 (local address 127.31.214.66:39449)
0928 06:25:06.913651 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:06.913664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:06.913731 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:06.914282 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:06.914283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:06.914285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:06.914359 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.914360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.915424 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.915426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.916669 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.916672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.917569 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.917570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.917650 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.917654 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:25:06.917706 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:25:06.917760 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280}
I20260928 06:25:06.918290 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.913439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:39449 (local address 127.31.214.65:35867)
0928 06:25:06.913563 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.913565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.914070 (+ 505us) server_negotiation.cc:407] Connection header received
0928 06:25:06.914100 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.914102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.914114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.914162 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:06.914436 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.915293 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.916811 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.917447 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.917525 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.917535 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.918169 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.918220 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 06:25:06.918577 11548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7985271c0489299726dd3e536171c" candidate_uuid: "c9ea9713c9084934ad55812cfa50fbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a02c2f09b740d49cf62a22dc5020ee" is_pre_election: true
I20260928 06:25:06.918748 11548 raft_consensus.cc:2454] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ea9713c9084934ad55812cfa50fbcc in term 0.
I20260928 06:25:06.919066 11619 leader_election.cc:302] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36a02c2f09b740d49cf62a22dc5020ee, c9ea9713c9084934ad55812cfa50fbcc; no voters:
I20260928 06:25:06.919358 11892 raft_consensus.cc:2781] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:25:06.919431 11892 raft_consensus.cc:482] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:25:06.919456 11892 raft_consensus.cc:3037] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:25:06.921325 11892 raft_consensus.cc:504] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } }
I20260928 06:25:06.921499 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.913836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:60671 (local address 127.31.214.67:44289)
0928 06:25:06.913886 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.913888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.913897 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:25:06.913925 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.913926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.913935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.914010 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:06.914464 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.915396 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.917578 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.918674 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.918732 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.918742 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.921367 (+ 2625us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.921426 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:25:06.921686 11548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7985271c0489299726dd3e536171c" candidate_uuid: "c9ea9713c9084934ad55812cfa50fbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a02c2f09b740d49cf62a22dc5020ee"
I20260928 06:25:06.921931 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.913457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:44289 (local address 127.31.214.66:60671)
0928 06:25:06.913677 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:06.913689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:06.913747 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:06.914137 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:06.914138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:06.914141 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:06.914300 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.914303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.915658 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.915659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.917461 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:06.917464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.918828 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.918829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:06.921241 (+ 2412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.921248 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:25:06.921757 (+ 509us) client_negotiation.cc:240] Negotiation successful
0928 06:25:06.921848 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":114,"threads_started":1}
I20260928 06:25:06.921897 11548 raft_consensus.cc:3037] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:25:06.922569 11892 leader_election.cc:288] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [CANDIDATE]: Term 1 election: Requested vote from peers 36a02c2f09b740d49cf62a22dc5020ee (127.31.214.65:35867), 719c0cfceaa94021985d6357ec1f1633 (127.31.214.67:44289)
I20260928 06:25:06.922538 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7985271c0489299726dd3e536171c" candidate_uuid: "c9ea9713c9084934ad55812cfa50fbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "719c0cfceaa94021985d6357ec1f1633" is_pre_election: true
I20260928 06:25:06.922708 11805 raft_consensus.cc:2454] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ea9713c9084934ad55812cfa50fbcc in term 0.
I20260928 06:25:06.922753 11806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7985271c0489299726dd3e536171c" candidate_uuid: "c9ea9713c9084934ad55812cfa50fbcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "719c0cfceaa94021985d6357ec1f1633"
I20260928 06:25:06.922812 11806 raft_consensus.cc:3037] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:25:06.923730 11806 raft_consensus.cc:2454] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ea9713c9084934ad55812cfa50fbcc in term 1.
I20260928 06:25:06.923918 11548 raft_consensus.cc:2454] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ea9713c9084934ad55812cfa50fbcc in term 1.
I20260928 06:25:06.923925 11619 leader_election.cc:302] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 719c0cfceaa94021985d6357ec1f1633, c9ea9713c9084934ad55812cfa50fbcc; no voters:
I20260928 06:25:06.924012 11892 raft_consensus.cc:2781] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:25:06.924185 11892 raft_consensus.cc:686] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [term 1 LEADER]: Becoming Leader. State: Replica: c9ea9713c9084934ad55812cfa50fbcc, State: Running, Role: LEADER
I20260928 06:25:06.924309 11892 consensus_queue.cc:237] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } }
I20260928 06:25:06.925249 11381 catalog_manager.cc:5495] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc reported cstate change: term changed from 0 to 1, leader changed from <none> to c9ea9713c9084934ad55812cfa50fbcc (127.31.214.66). New cstate: current_term: 1 leader_uuid: "c9ea9713c9084934ad55812cfa50fbcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ea9713c9084934ad55812cfa50fbcc" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 41583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 } health_report { overall_health: UNKNOWN } } }
I20260928 06:25:06.949824 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.945606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.31.214.66:41583)
0928 06:25:06.945669 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:06.945671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:06.945684 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:25:06.945765 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:06.945767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:06.945795 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:06.945909 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:25:06.946122 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.946966 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.948412 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:06.948776 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:06.948855 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:06.949186 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:25:06.949353 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:25:06.949438 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:06.949705 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 06:25:06.949731 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:25:06.952307 11548 raft_consensus.cc:1261] T 90f7985271c0489299726dd3e536171c P 36a02c2f09b740d49cf62a22dc5020ee [term 1 FOLLOWER]: Refusing update from remote peer c9ea9713c9084934ad55812cfa50fbcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:25:06.952469 11806 raft_consensus.cc:1261] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [term 1 FOLLOWER]: Refusing update from remote peer c9ea9713c9084934ad55812cfa50fbcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:25:06.952713 11895 consensus_queue.cc:1035] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a02c2f09b740d49cf62a22dc5020ee" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:25:06.952903 11895 consensus_queue.cc:1035] T 90f7985271c0489299726dd3e536171c P c9ea9713c9084934ad55812cfa50fbcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "719c0cfceaa94021985d6357ec1f1633" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:25:06.958740 11898 mvcc.cc:205] Tried to move back new op lower bound from 7334202191672385536 to 7334202191562792960. Current Snapshot: MvccSnapshot[applied={T|T < 7334202191672385536}]
I20260928 06:25:06.959760 11901 mvcc.cc:205] Tried to move back new op lower bound from 7334202191672385536 to 7334202191562792960. Current Snapshot: MvccSnapshot[applied={T|T < 7334202191672385536}]
I20260928 06:25:06.962482 11899 mvcc.cc:205] Tried to move back new op lower bound from 7334202191672385536 to 7334202191562792960. Current Snapshot: MvccSnapshot[applied={T|T < 7334202191672385536}]
Sep 28 06:25:06 dist-test-slave-q03f krb5kdc[10906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576706, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:06.965523 11382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:25:06.965626 11382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:25:06.966589 11382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 046373ea17584c739b0be901555e7091: Sending DeleteTablet for 3 replicas of tablet 90f7985271c0489299726dd3e536171c
I20260928 06:25:06.967119 11665 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f7985271c0489299726dd3e536171c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:25:06 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:44218
I20260928 06:25:06.967443 11528 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f7985271c0489299726dd3e536171c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:25:06 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:38830
I20260928 06:25:06.967483 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 11463
I20260928 06:25:06.968117 11800 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f7985271c0489299726dd3e536171c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:25:06 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:37858
I20260928 06:25:06.974745 11910 tablet_replica.cc:330] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: stopping tablet replica
W20260928 06:25:06.976630 11367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:35867 (error 108)
I20260928 06:25:06.977304 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 11600
I20260928 06:25:06.986068 11910 raft_consensus.cc:2229] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:25:06.986296 11910 raft_consensus.cc:2258] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:25:06.986930 11910 ts_tablet_manager.cc:1905] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:25:06.988682 11910 ts_tablet_manager.cc:1918] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:25:06.988772 11910 log.cc:1196] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/90f7985271c0489299726dd3e536171c
I20260928 06:25:06.989459 11910 ts_tablet_manager.cc:1939] T 90f7985271c0489299726dd3e536171c P 719c0cfceaa94021985d6357ec1f1633: Deleting consensus metadata
I20260928 06:25:06.990087 11367 catalog_manager.cc:4841] TS 719c0cfceaa94021985d6357ec1f1633 (127.31.214.67:44289): tablet 90f7985271c0489299726dd3e536171c (table test-table [id=eacb5eec780c4da2a1887421b6de9067]) successfully deleted
I20260928 06:25:06.994385 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:06.993670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:35867 (local address 127.0.0.1:38842)
0928 06:25:06.993745 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:06.993902 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:35867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 06:25:06.995265 11367 catalog_manager.cc:4584] TS 36a02c2f09b740d49cf62a22dc5020ee (127.31.214.65:35867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90f7985271c0489299726dd3e536171c: Network error: Client connection negotiation failed: client connection to 127.31.214.65:35867: connect: Connection refused (error 111)
I20260928 06:25:06.998198 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 11737
I20260928 06:25:07.002115 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:07.001985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:41583 (local address 127.0.0.1:44244)
0928 06:25:07.002046 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:07.002076 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.66:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:25:07.007602 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 11350
I20260928 06:25:07.024109 32601 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:25:07.379279 32601 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:25:07.379391 32601 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:25:07.724560 32601 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:25:07.724 UTC [10936] LOG: received smart shutdown request
2026-09-28 06:25:07.729 UTC [10936] LOG: background worker "logical replication launcher" (PID 10945) exited with exit code 1
2026-09-28 06:25:07.729 UTC [10940] LOG: shutting down
2026-09-28 06:25:07.945 UTC [10936] LOG: database system is shut down
2026-09-28T06:25:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51887 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:25:08 dist-test-slave-q03f krb5kdc[11920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:25:08 dist-test-slave-q03f krb5kdc[11920](info): set up 2 sockets
Sep 28 06:25:08 dist-test-slave-q03f krb5kdc[11920](info): commencing operation
krb5kdc: starting...
W20260928 06:25:09.194205 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:25:09 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:25:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.214.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.214.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/rangeradmin_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/rangeradmin_127.31.214.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.214.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.214.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/rangerlookup_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/rangerlookup_127.31.214.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.214.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.214.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/HTTP_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/HTTP_127.31.214.84@KRBTEST.COM.keytab.
I20260928 06:25:09.384248 32601 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDYGmsK/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/postgres -l logfile start
2026-09-28 06:25:12.372 UTC [11950] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:25:12.372 UTC [11950] LOG: listening on IPv4 address "127.31.214.84", port 50599
2026-09-28 06:25:12.376 UTC [11950] LOG: listening on Unix socket "/tmp/.s.PGSQL.50599"
2026-09-28 06:25:12.412 UTC [11953] LOG: database system was shut down at 2026-09-28 06:25:10 UTC
2026-09-28 06:25:12.417 UTC [11950] LOG: database system is ready to accept connections
I20260928 06:25:13.466845 32601 mini_postgres.cc:96] Postgres bound to 50599
2026-09-28 06:25:13.473 UTC [11961] FATAL: database "slave" does not exist
127.31.214.84:50599 - accepting connections
I20260928 06:25:13.474362 32601 mini_ranger.cc:161] Starting Ranger...
I20260928 06:25:13.491125 32601 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:25:14.072278 32601 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:25:14.072369 32601 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-admin
2026-09-28 06:25:14,458 [I] DB FLAVOR :POSTGRES
2026-09-28 06:25:14,459 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:25:14,459 [I] Checking connection..
2026-09-28 06:25:14,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:25:14,826 [I] Checking connection passed.
2026-09-28 06:25:14,826 [I] --------- Verifying version history table ---------
2026-09-28 06:25:14,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:25:15,185 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:25:15,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:25:15,474 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:25:15,475 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:25:15,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:25:15.725 UTC [12025] WARNING: there is no transaction in progress
2026-09-28 06:25:15,733 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:25:15,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:25:15,998 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:25:15,999 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:25:15,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:25:16,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:25:16,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q03f.c.gcp-upstream.internal','N') ;"
2026-09-28 06:25:16,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:25:17,045 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:25:17,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:25:17,301 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:25:17,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:25:17,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:25:17,838 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:25:17,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:25:18.941 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:18.962 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:18.981 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:18.999 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.018 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.040 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.108 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.116 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.133 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.145 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.158 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.174 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.185 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.194 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.203 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:19.212 UTC [12133] WARNING: there is no transaction in progress
2026-09-28 06:25:20,011 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:25:20,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:25:20,262 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:25:20,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:25:20,556 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:25:20,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:25:20,791 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:25:20,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:25:21,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:25:21,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q03f.c.gcp-upstream.internal';"
2026-09-28 06:25:21,535 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:25:21,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:25:21,802 [I] DB_PATCHES have already been applied
I20260928 06:25:21.807649 32601 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDYGmsK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:25:21.807716 32601 mini_ranger.cc:193] Using host: 127.31.214.84
I20260928 06:25:21.811056 32601 mini_ranger.cc:227] Ranger admin URL: http://127.31.214.84:48127
Sep 28, 2026 6:25:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:25:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:25:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:25:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:25:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:25:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:25:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:25:22.935133 32601 mini_ranger.cc:160] Time spent starting Ranger: real 9.461s user 0.000s sys 0.010s
Sep 28, 2026 6:25:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:25:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 6:25:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:25:45 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:25:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:25:46.634423 32601 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.214.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.214.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/rangerkms_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.214.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/rangerkms_127.31.214.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.214.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.214.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.214.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/HTTP_127.31.214.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.214.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/HTTP_127.31.214.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:25:46 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576746, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:25:46.725598 32601 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:25:46.735612 32601 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:25:47.778580 32601 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:25:47.778656 32601 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-kms
I20260928 06:25:47.778669 32601 mini_ranger_kms.cc:227] Using postgres at 127.31.214.84:50599
2026-09-28 06:25:47,955 [I] DB FLAVOR :POSTGRES
2026-09-28 06:25:47,955 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:25:47,956 [I] Checking connection
2026-09-28 06:25:47,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:25:48,220 [I] connection success
2026-09-28 06:25:48,220 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:25:48,221 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:25:48,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:25:48,487 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:25:48,487 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:25:48,487 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:25:48,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.214.84:50599/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:25:48,806 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:25:49.330567 32601 mini_ranger_kms.cc:310] Created kms service
I20260928 06:25:49.547953 32601 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:25:49.598129 32601 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:25:49 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576746, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:50.044056 32601 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:25:50.044205 32601 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDYGmsK/build/release/bin/postgresql.jar:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDYGmsK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:25:50.044238 32601 mini_ranger_kms.cc:240] Using host: 127.31.214.84
I20260928 06:25:50.048233 32601 mini_ranger_kms.cc:276] Ranger KMS PID: 12314
I20260928 06:25:50.048338 32601 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.214.84:60825
Sep 28, 2026 6:25:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:25:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:25:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:25:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:25:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:25:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:25:52.412638 32601 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.687s user 0.013s sys 0.147s
I20260928 06:25:52.412758 32601 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:25:52.412820 32601 mini_ranger_kms.cc:401] 127.31.214.84:60825/kms/v1/keys
Sep 28 06:25:52 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790576752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.214.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:25:52 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.214.84@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
Sep 28 06:25:53 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.214.84@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
Sep 28 06:25:55 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:25:55.757326 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:34407
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:34407
--ranger_config_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:60825/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:25:55.846840 12364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:55.847117 12364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:55.847153 12364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:55.848464 12364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:25:55.848546 12364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:55.848572 12364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:55.848593 12364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:25:55.848621 12364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:25:55.850585 12364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.214.84:60825/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:34407
--ranger_config_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:34407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:55.850929 12364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:55.851199 12364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:55.856900 12372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:55.856887 12370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:55.856886 12369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:55.857211 12364 server_base.cc:1034] running on GCE node
I20260928 06:25:55.857704 12364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:55.858098 12364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:55.859267 12364 hybrid_clock.cc:648] HybridClock initialized: now 1790576755859245 us; error 42 us; skew 500 ppm
Sep 28 06:25:55 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:55.862054 12364 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:25:55.862576 12364 webserver.cc:460] Webserver started at http://127.31.214.126:33157/ using document root <none> and password file <none>
I20260928 06:25:55.862848 12364 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:55.862910 12364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:25:55.863054 12364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:25:55 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576755, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:55.922911 12364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "4a8b7739704d4563bc2e56771b84ae5e"
format_stamp: "Formatted at 2026-09-28 06:25:55 on dist-test-slave-q03f"
server_key: "c5a1513e8a81584f393db1cb5129c577"
server_key_iv: "84640f7e835b886c88072cf947d9d225"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:55.923336 12364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "4a8b7739704d4563bc2e56771b84ae5e"
format_stamp: "Formatted at 2026-09-28 06:25:55 on dist-test-slave-q03f"
server_key: "c5a1513e8a81584f393db1cb5129c577"
server_key_iv: "84640f7e835b886c88072cf947d9d225"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:55.924916 12364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:25:55.937461 12379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:55.937770 12364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:25:55.937899 12364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "4a8b7739704d4563bc2e56771b84ae5e"
format_stamp: "Formatted at 2026-09-28 06:25:55 on dist-test-slave-q03f"
server_key: "c5a1513e8a81584f393db1cb5129c577"
server_key_iv: "84640f7e835b886c88072cf947d9d225"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:55.938031 12364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:55.968039 12364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:55.970825 12364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:55.971011 12364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:55.976253 12364 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:34407
I20260928 06:25:55.976293 12431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:34407 every 8 connection(s)
I20260928 06:25:55.976714 12364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:25:55.977600 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:25:55.980104 12432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Bootstrap starting.
I20260928 06:25:55.980808 12432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Neither blocks nor log segments found. Creating new log.
I20260928 06:25:55.981118 12432 log.cc:824] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:25:55.982015 12432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: No bootstrap required, opened a new log
I20260928 06:25:55.984778 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 12364
I20260928 06:25:55.984887 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:25:55.985105 32601 external_mini_cluster.cc:1428] Setting key ef8b7b14a0ab726513179be17b03ef5d
I20260928 06:25:55.985028 12432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } }
I20260928 06:25:55.985247 12432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:25:55.985353 12432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8b7739704d4563bc2e56771b84ae5e, State: Initialized, Role: FOLLOWER
I20260928 06:25:55.985554 12432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } }
I20260928 06:25:55.985674 12432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:25:55.985728 12432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:25:55.985790 12432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:25:55.986861 12432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } }
I20260928 06:25:55.986991 12432 leader_election.cc:302] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a8b7739704d4563bc2e56771b84ae5e; no voters:
I20260928 06:25:55.987222 12432 leader_election.cc:288] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:25:55.987286 12437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:25:55.987550 12437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 1 LEADER]: Becoming Leader. State: Replica: 4a8b7739704d4563bc2e56771b84ae5e, State: Running, Role: LEADER
I20260928 06:25:55.987622 12432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:25:55.987706 12437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } }
I20260928 06:25:55.988130 12437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a8b7739704d4563bc2e56771b84ae5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } } }
I20260928 06:25:55.988276 12437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: This master's current role is: LEADER
I20260928 06:25:55.988451 12437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a8b7739704d4563bc2e56771b84ae5e. Latest consensus state: current_term: 1 leader_uuid: "4a8b7739704d4563bc2e56771b84ae5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } } }
I20260928 06:25:55.988546 12437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: This master's current role is: LEADER
I20260928 06:25:55.989197 12432 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:25:55.992794 12446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:25:55.993808 12446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:25:55.995991 12446 catalog_manager.cc:1347] Generated new cluster ID: 7d644713368346d0940a2c0f3722b459
I20260928 06:25:55.996124 12446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:25:55 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:56.001376 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:55.985740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.31.214.126:34407)
0928 06:25:55.985990 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:55.986003 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:55.986155 (+ 152us) server_negotiation.cc:407] Connection header received
0928 06:25:55.986323 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:55.986328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:55.986377 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:55.986528 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:55.987395 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:55.988967 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:55.991143 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:55.992760 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:55.992903 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:55.996913 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:55.996919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:55.996919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:55.996946 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:55.999230 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:55.999819 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:55.999823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:55.999824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:55.999971 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:56.000212 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:56.000213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:56.000213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:56.000638 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:56.000811 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:56.000999 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:25:56.001139 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260928 06:25:56.015295 12446 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:25:56.016249 12446 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:25:56.031623 12446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Generated new TSK 0
I20260928 06:25:56.032006 12446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:25:56.858786 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:60825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:34407
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:25:56.971939 12477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:56.972193 12477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:56.972232 12477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:56.973533 12477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:56.973606 12477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:56.973652 12477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:25:56.975955 12477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.214.84:60825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:34407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:56.976569 12477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:56.976926 12477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:56.984311 12482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:56.984594 12483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:56.985998 12477 server_base.cc:1034] running on GCE node
W20260928 06:25:56.986155 12485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:56.986423 12477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:56.986694 12477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:56.987893 12477 hybrid_clock.cc:648] HybridClock initialized: now 1790576756987876 us; error 27 us; skew 500 ppm
Sep 28 06:25:56 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576756, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:56.990976 12477 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:25:56.991572 12477 webserver.cc:460] Webserver started at http://127.31.214.65:45009/ using document root <none> and password file <none>
I20260928 06:25:56.991837 12477 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:56.991883 12477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:25:56.992015 12477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:25:56 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576756, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:57.007126 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "0284c66eeea3454ca007a9e91c75f3b8"
format_stamp: "Formatted at 2026-09-28 06:25:56 on dist-test-slave-q03f"
server_key: "6f9c0771637f1088e9f95594f9853b62"
server_key_iv: "893446695588142bb9c243244ea09f68"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.007658 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "0284c66eeea3454ca007a9e91c75f3b8"
format_stamp: "Formatted at 2026-09-28 06:25:56 on dist-test-slave-q03f"
server_key: "6f9c0771637f1088e9f95594f9853b62"
server_key_iv: "893446695588142bb9c243244ea09f68"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.009836 12477 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:25:57.029567 12493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.030624 12477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:25:57.030788 12477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "0284c66eeea3454ca007a9e91c75f3b8"
format_stamp: "Formatted at 2026-09-28 06:25:56 on dist-test-slave-q03f"
server_key: "6f9c0771637f1088e9f95594f9853b62"
server_key_iv: "893446695588142bb9c243244ea09f68"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.030895 12477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:57.059561 12477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:57.062232 12477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:57.062457 12477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:57.062798 12477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:25:57.063211 12477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:25:57.063299 12477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.063395 12477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:25:57.063468 12477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.071767 12477 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:46609
I20260928 06:25:57.072315 12477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:25:57.073521 12607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:46609 every 8 connection(s)
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576756, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:57.081014 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 12477
I20260928 06:25:57.081099 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:25:57.081333 32601 external_mini_cluster.cc:1428] Setting key 45b62d5b49553aa2c3d37fbed3af1148
I20260928 06:25:57.087949 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:57.073199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:35987 (local address 127.31.214.126:34407)
0928 06:25:57.073331 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:57.073333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:57.073543 (+ 210us) server_negotiation.cc:407] Connection header received
0928 06:25:57.074382 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:57.074383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:57.074400 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:57.074499 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:57.075347 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.076216 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.077786 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.078143 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.078195 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:57.084037 (+ 5842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:57.084039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:57.084040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:57.084049 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:57.086051 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:57.086626 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:57.086628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:57.086628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:57.086693 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:57.087012 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:57.087013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:57.087013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:57.087164 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:57.087213 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:57.087728 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 06:25:57.087838 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260928 06:25:57.088871 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:57.073199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:34407 (local address 127.31.214.65:35987)
0928 06:25:57.073405 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:57.073448 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:57.074233 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:57.074595 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:57.074599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:57.074879 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:25:57.075213 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:57.075220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.076349 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.076350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:57.077673 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:57.077676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.078310 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.078311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:57.078458 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:57.079088 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:25:57.079099 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:25:57.082054 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:25:57.086161 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:57.086169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:57.086173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:57.086526 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:57.086766 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:57.086766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:57.086767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:57.086929 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:57.087266 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:25:57.087274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:25:57.087478 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 06:25:57.088337 (+ 859us) client_negotiation.cc:240] Negotiation successful
0928 06:25:57.088526 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260928 06:25:57.089679 12608 heartbeater.cc:347] Connected to a master server at 127.31.214.126:34407
I20260928 06:25:57.089780 12608 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:57.090036 12608 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
I20260928 06:25:57.090690 12396 ts_manager.cc:194] Registered new tserver with Master: 0284c66eeea3454ca007a9e91c75f3b8 (127.31.214.65:46609)
I20260928 06:25:57.091602 12396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:35987
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:25:57.137744 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:60825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:34407
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790576757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 06:25:57.288375 12614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:57.288753 12614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:57.288833 12614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:57.290681 12614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:57.290812 12614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:57.290905 12614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:25:57.293099 12614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.214.84:60825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:34407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:57.293609 12614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:57.294013 12614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:57.300810 12622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:57.301186 12621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:57.301440 12624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:57.307268 12614 server_base.cc:1034] running on GCE node
I20260928 06:25:57.307669 12614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:57.307991 12614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:57.315328 12614 hybrid_clock.cc:648] HybridClock initialized: now 1790576757315298 us; error 57 us; skew 500 ppm
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:57.318851 12614 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:25:57.319388 12614 webserver.cc:460] Webserver started at http://127.31.214.66:40435/ using document root <none> and password file <none>
I20260928 06:25:57.319638 12614 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:57.319684 12614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:25:57.319808 12614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:57.340545 12614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "009db381c0aa4b83a7dfe1cb943bcc70"
format_stamp: "Formatted at 2026-09-28 06:25:57 on dist-test-slave-q03f"
server_key: "0b78da83a9c8b4d5b1ee68dd9ae71a44"
server_key_iv: "3f08c0a52378c1a70aa553ce354e3e37"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.342020 12614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "009db381c0aa4b83a7dfe1cb943bcc70"
format_stamp: "Formatted at 2026-09-28 06:25:57 on dist-test-slave-q03f"
server_key: "0b78da83a9c8b4d5b1ee68dd9ae71a44"
server_key_iv: "3f08c0a52378c1a70aa553ce354e3e37"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.344750 12614 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:25:57.361652 12631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.361882 12614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:25:57.362020 12614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "009db381c0aa4b83a7dfe1cb943bcc70"
format_stamp: "Formatted at 2026-09-28 06:25:57 on dist-test-slave-q03f"
server_key: "0b78da83a9c8b4d5b1ee68dd9ae71a44"
server_key_iv: "3f08c0a52378c1a70aa553ce354e3e37"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.362162 12614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:57.376427 12614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:57.379168 12614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:57.379375 12614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:57.379665 12614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:25:57.380054 12614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:25:57.380106 12614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.380146 12614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:25:57.380178 12614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.389967 12614 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:36783
I20260928 06:25:57.390462 12614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:25:57.397562 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 12614
I20260928 06:25:57.397672 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:25:57.397890 32601 external_mini_cluster.cc:1428] Setting key 2152f0a983e29eff9bc442f7b0cd306e
I20260928 06:25:57.394244 12745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:36783 every 8 connection(s)
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:57.410984 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:57.395726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:54019 (local address 127.31.214.126:34407)
0928 06:25:57.395785 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:57.395788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:57.396371 (+ 583us) server_negotiation.cc:407] Connection header received
0928 06:25:57.400509 (+ 4138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:57.400510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:57.400535 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:57.400574 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:57.401555 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.402462 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.404324 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.404665 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.404709 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:57.407472 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:57.407473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:57.407474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:57.407482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:57.409278 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:57.409812 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:57.409813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:57.409813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:57.409877 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:57.410202 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:57.410203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:57.410203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:57.410345 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:57.410388 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:57.410779 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 06:25:57.410878 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:25:57.411832 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:57.395188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:34407 (local address 127.31.214.66:54019)
0928 06:25:57.396197 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:57.396247 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:57.397249 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:57.400827 (+ 3578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:57.400832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:57.401099 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:25:57.401436 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:57.401445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.402858 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.402860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:57.404211 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:57.404214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.404840 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.404842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:57.404991 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:57.405645 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:25:57.405656 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:25:57.407334 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:25:57.409380 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:57.409390 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:57.409393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:57.409703 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:57.409983 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:57.409984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:57.409984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:57.410127 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:57.410436 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:25:57.410443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:25:57.410615 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 06:25:57.411373 (+ 758us) client_negotiation.cc:240] Negotiation successful
0928 06:25:57.411564 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":273,"threads_started":1}
I20260928 06:25:57.412618 12746 heartbeater.cc:347] Connected to a master server at 127.31.214.126:34407
I20260928 06:25:57.412720 12746 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:57.412927 12746 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
I20260928 06:25:57.413317 12396 ts_manager.cc:194] Registered new tserver with Master: 009db381c0aa4b83a7dfe1cb943bcc70 (127.31.214.66:36783)
I20260928 06:25:57.413837 12396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:54019
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:25:57.449258 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:60825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:34407
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:25:57.586515 12751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:57.586951 12751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:57.587054 12751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:57.589090 12751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:57.589280 12751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:57.589375 12751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:25:57.591933 12751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.214.84:60825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:34407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:57.592496 12751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:57.592903 12751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:57.600080 12756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:57.600409 12757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:57.600469 12759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:57.600917 12751 server_base.cc:1034] running on GCE node
I20260928 06:25:57.601210 12751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:57.601526 12751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:57.602731 12751 hybrid_clock.cc:648] HybridClock initialized: now 1790576757602701 us; error 43 us; skew 500 ppm
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:57.606665 12751 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:25:57.607273 12751 webserver.cc:460] Webserver started at http://127.31.214.67:45561/ using document root <none> and password file <none>
I20260928 06:25:57.607565 12751 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:57.607643 12751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:25:57.607806 12751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:57.626801 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "619106b547f14daa837f1e2569a2463a"
format_stamp: "Formatted at 2026-09-28 06:25:57 on dist-test-slave-q03f"
server_key: "cc7865cd4b7c2b3951aecca48d6a7b5e"
server_key_iv: "d8f8eb7fd0cb1d6c95c00523e79bdbcd"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.627310 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "619106b547f14daa837f1e2569a2463a"
format_stamp: "Formatted at 2026-09-28 06:25:57 on dist-test-slave-q03f"
server_key: "cc7865cd4b7c2b3951aecca48d6a7b5e"
server_key_iv: "d8f8eb7fd0cb1d6c95c00523e79bdbcd"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.629027 12751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:25:57.644253 12766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.644568 12751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:25:57.644766 12751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "619106b547f14daa837f1e2569a2463a"
format_stamp: "Formatted at 2026-09-28 06:25:57 on dist-test-slave-q03f"
server_key: "cc7865cd4b7c2b3951aecca48d6a7b5e"
server_key_iv: "d8f8eb7fd0cb1d6c95c00523e79bdbcd"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.644889 12751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:57.673892 12751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:57.676611 12751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:57.676831 12751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:57.677127 12751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:25:57.677536 12751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:25:57.677591 12751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.677624 12751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:25:57.677647 12751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.685318 12751 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:41911
I20260928 06:25:57.685705 12751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:25:57.686305 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 12751
I20260928 06:25:57.686424 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:25:57.686637 32601 external_mini_cluster.cc:1428] Setting key e6524fe7615601137b84e68ea7405174
I20260928 06:25:57.691118 12879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:41911 every 8 connection(s)
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:57.700214 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:57.686928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:48911 (local address 127.31.214.126:34407)
0928 06:25:57.686978 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:57.686980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:57.690292 (+ 3312us) server_negotiation.cc:407] Connection header received
0928 06:25:57.690858 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:57.690859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:57.690913 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:57.690972 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:57.691779 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.692280 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.693322 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.693534 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.693615 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:57.696437 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:57.696439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:57.696439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:57.696449 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:57.698406 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:57.698995 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:57.698997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:57.698997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:57.699062 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:57.699371 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:57.699372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:57.699373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:57.699526 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:57.699572 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:57.700015 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 06:25:57.700111 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:25:57.701468 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:57.686563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:34407 (local address 127.31.214.67:48911)
0928 06:25:57.690130 (+ 3567us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:57.690179 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:57.690738 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:57.691112 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:57.691117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:57.691450 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:25:57.691662 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:57.691667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.692389 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.692390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:57.693223 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:57.693225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:57.693639 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:57.693641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:57.693801 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:57.694496 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:25:57.694507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:25:57.696310 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:25:57.698523 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:57.698534 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:57.698537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:57.698879 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:57.699137 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:57.699138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:57.699138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:57.699285 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:57.699622 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:25:57.699630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:25:57.699815 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 06:25:57.700386 (+ 571us) client_negotiation.cc:240] Negotiation successful
0928 06:25:57.701190 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3505,"thread_start_us":90,"threads_started":1}
I20260928 06:25:57.701975 12880 heartbeater.cc:347] Connected to a master server at 127.31.214.126:34407
I20260928 06:25:57.702085 12880 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:57.702301 12880 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
I20260928 06:25:57.702711 12396 ts_manager.cc:194] Registered new tserver with Master: 619106b547f14daa837f1e2569a2463a (127.31.214.67:41911)
I20260928 06:25:57.703266 12396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:48911
I20260928 06:25:57.712008 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:25:57.712776 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 12477
I20260928 06:25:57.719480 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 12614
I20260928 06:25:57.726475 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 12751
I20260928 06:25:57.733052 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 12364
I20260928 06:25:57.739346 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:34407
--webserver_interface=127.31.214.126
--webserver_port=33157
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:34407
--ranger_config_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:60825/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.214.84:60825/kms with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:25:57.828477 12883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:57.828722 12883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:57.828747 12883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:57.830086 12883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:25:57.830216 12883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:57.830240 12883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:57.830255 12883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:25:57.830269 12883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:25:57.831940 12883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.214.84:60825/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:34407
--ranger_config_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:34407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:57.832158 12883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:57.832363 12883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:57.837747 12891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:57.837790 12889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:57.837781 12888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:57.837824 12883 server_base.cc:1034] running on GCE node
I20260928 06:25:57.838192 12883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:57.838529 12883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:57.839751 12883 hybrid_clock.cc:648] HybridClock initialized: now 1790576757839739 us; error 41 us; skew 500 ppm
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:57.842813 12883 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:25:57.843307 12883 webserver.cc:460] Webserver started at http://127.31.214.126:33157/ using document root <none> and password file <none>
I20260928 06:25:57.843544 12883 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:57.843622 12883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:25:57 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576757, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:57.900696 12883 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:25:57.901504 12899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:57.901773 12883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:25:57.901903 12883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "4a8b7739704d4563bc2e56771b84ae5e"
format_stamp: "Formatted at 2026-09-28 06:25:55 on dist-test-slave-q03f"
server_key: "c5a1513e8a81584f393db1cb5129c577"
server_key_iv: "84640f7e835b886c88072cf947d9d225"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:57.902453 12883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:57.924824 12883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:57.927181 12883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:57.927336 12883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:57.932291 12883 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:34407
I20260928 06:25:57.932358 12951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:34407 every 8 connection(s)
I20260928 06:25:57.932758 12883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:25:57.934253 12952 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:25:57.935055 12952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Bootstrap starting.
I20260928 06:25:57.936915 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 12883
I20260928 06:25:57.937362 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:60825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:46609
--local_ip_for_outbound_sockets=127.31.214.65
--tserver_master_addrs=127.31.214.126:34407
--webserver_port=45009
--webserver_interface=127.31.214.65
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.214.84:60825/kms with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:25:57.944159 12952 log.cc:824] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:25:57.946264 12952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:25:57.946580 12952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Bootstrap complete.
I20260928 06:25:57.949474 12952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } }
I20260928 06:25:57.949792 12952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8b7739704d4563bc2e56771b84ae5e, State: Initialized, Role: FOLLOWER
I20260928 06:25:57.950043 12952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } }
I20260928 06:25:57.950167 12952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:25:57.950229 12952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:25:57.950388 12952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:25:57.951805 12952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } }
I20260928 06:25:57.951941 12952 leader_election.cc:302] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a8b7739704d4563bc2e56771b84ae5e; no voters:
I20260928 06:25:57.952176 12952 leader_election.cc:288] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:25:57.952219 12956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:25:57.952417 12956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [term 2 LEADER]: Becoming Leader. State: Replica: 4a8b7739704d4563bc2e56771b84ae5e, State: Running, Role: LEADER
I20260928 06:25:57.952486 12952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:25:57.952608 12956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } }
I20260928 06:25:57.953020 12957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4a8b7739704d4563bc2e56771b84ae5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } } }
I20260928 06:25:57.953022 12958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a8b7739704d4563bc2e56771b84ae5e. Latest consensus state: current_term: 2 leader_uuid: "4a8b7739704d4563bc2e56771b84ae5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8b7739704d4563bc2e56771b84ae5e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34407 } } }
I20260928 06:25:57.953156 12957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: This master's current role is: LEADER
I20260928 06:25:57.953194 12958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e [sys.catalog]: This master's current role is: LEADER
I20260928 06:25:57.953775 12964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:25:57.954241 12952 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:25:57.954440 12964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:25:57.954615 12964 catalog_manager.cc:1259] Loaded cluster ID: 7d644713368346d0940a2c0f3722b459
I20260928 06:25:57.954656 12964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:25:57.955806 12964 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:25:57.956012 12964 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Loaded TSK: 0
I20260928 06:25:57.956241 12964 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:25:58.034303 12954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:58.034583 12954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:58.034624 12954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:58.036015 12954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:58.036080 12954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:58.036124 12954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:25:58.038465 12954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.214.84:60825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:46609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=45009
--tserver_master_addrs=127.31.214.126:34407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:58.038996 12954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:58.039378 12954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:58.045667 12989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:58.045863 12991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:58.046074 12988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:58.046232 12954 server_base.cc:1034] running on GCE node
I20260928 06:25:58.046478 12954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:58.046824 12954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:58.048051 12954 hybrid_clock.cc:648] HybridClock initialized: now 1790576758048021 us; error 48 us; skew 500 ppm
Sep 28 06:25:58 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:58.050905 12954 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:25:58.051435 12954 webserver.cc:460] Webserver started at http://127.31.214.65:45009/ using document root <none> and password file <none>
I20260928 06:25:58.052325 12954 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:58.052425 12954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:25:58 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:58.072865 12954 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.000s
I20260928 06:25:58.074051 13000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:58.074281 12954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:25:58.074369 12954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "0284c66eeea3454ca007a9e91c75f3b8"
format_stamp: "Formatted at 2026-09-28 06:25:56 on dist-test-slave-q03f"
server_key: "6f9c0771637f1088e9f95594f9853b62"
server_key_iv: "893446695588142bb9c243244ea09f68"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:58.074855 12954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:58.103157 12954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:58.105859 12954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:58.106072 12954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:58.106359 12954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:25:58.107030 12954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:25:58.107116 12954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:58.107177 12954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:25:58.107211 12954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:58.114435 12954 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:46609
I20260928 06:25:58.115020 12954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:25:58.122202 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 12954
I20260928 06:25:58.122593 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:60825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:36783
--local_ip_for_outbound_sockets=127.31.214.66
--tserver_master_addrs=127.31.214.126:34407
--webserver_port=40435
--webserver_interface=127.31.214.66
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.214.84:60825/kms with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:25:58.124612 13113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:46609 every 8 connection(s)
Sep 28 06:25:58 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:58.143117 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:58.125834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:34407 (local address 127.31.214.65:33199)
0928 06:25:58.126932 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:58.126990 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:58.127765 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:58.128534 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:58.128539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:58.128819 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:25:58.129125 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:58.129131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.131154 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.131155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:58.132607 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:58.132611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.133726 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.133728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:58.133901 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:58.134596 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:25:58.134608 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:25:58.136275 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:25:58.141151 (+ 4876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:58.141160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:58.141163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:58.141481 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:58.141787 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:58.141788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:58.141788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:58.141913 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:58.142440 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:25:58.142445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:25:58.142597 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 06:25:58.142689 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:25:58.142868 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"thread_start_us":98,"threads_started":1}
I20260928 06:25:58.143959 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:58.125934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:33199 (local address 127.31.214.126:34407)
0928 06:25:58.126825 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:58.126836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:58.127955 (+ 1119us) server_negotiation.cc:407] Connection header received
0928 06:25:58.128128 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:58.128135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:58.128202 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:58.128344 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:58.129276 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.131005 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.132724 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.133469 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.133589 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:58.136441 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:58.136447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:58.136447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:58.136461 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:58.141033 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:58.141572 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:58.141576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:58.141577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:58.141705 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:58.142009 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:58.142010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:58.142011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:58.142335 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:58.142413 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:58.143586 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0928 06:25:58.143714 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":112,"threads_started":1}
I20260928 06:25:58.145148 13114 heartbeater.cc:347] Connected to a master server at 127.31.214.126:34407
I20260928 06:25:58.145246 13114 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:58.145442 13114 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
W20260928 06:25:58.146865 13114 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:34407 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:25:58.146955 13114 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:58.147032 13114 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
I20260928 06:25:58.147461 13114 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:58.147550 13114 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
W20260928 06:25:58.147990 13114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:25:58.230577 13116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:58.230851 13116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:58.230901 13116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:58.232432 13116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:58.232519 13116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:58.232571 13116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:25:58.234582 13116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.214.84:60825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:36783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=40435
--tserver_master_addrs=127.31.214.126:34407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:58.234907 13116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:58.235206 13116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:58.241268 13126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:58.241289 13124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:58.241411 13123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:58.242111 13116 server_base.cc:1034] running on GCE node
I20260928 06:25:58.242424 13116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:58.242743 13116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:58.243932 13116 hybrid_clock.cc:648] HybridClock initialized: now 1790576758243899 us; error 50 us; skew 500 ppm
Sep 28 06:25:58 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:58.247193 13116 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:25:58.247864 13116 webserver.cc:460] Webserver started at http://127.31.214.66:40435/ using document root <none> and password file <none>
I20260928 06:25:58.248215 13116 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:58.248373 13116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:25:58 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:58.264599 13116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:25:58.265779 13134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:58.266052 13116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:25:58.266149 13116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "009db381c0aa4b83a7dfe1cb943bcc70"
format_stamp: "Formatted at 2026-09-28 06:25:57 on dist-test-slave-q03f"
server_key: "0b78da83a9c8b4d5b1ee68dd9ae71a44"
server_key_iv: "3f08c0a52378c1a70aa553ce354e3e37"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:58.266655 13116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:58.283751 13116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:58.286363 13116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:58.286624 13116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:58.286909 13116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:25:58.287374 13116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:25:58.287467 13116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:58.287554 13116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:25:58.287611 13116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:58.296517 13116 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:36783
I20260928 06:25:58.297139 13116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:25:58.298755 13247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:36783 every 8 connection(s)
I20260928 06:25:58.301661 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 13116
I20260928 06:25:58.302043 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.214.84:60825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:41911
--local_ip_for_outbound_sockets=127.31.214.67
--tserver_master_addrs=127.31.214.126:34407
--webserver_port=45561
--webserver_interface=127.31.214.67
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.214.84:60825/kms with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:25:58 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:58.309494 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:58.298422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:34407 (local address 127.31.214.66:45045)
0928 06:25:58.298952 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:58.298999 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:58.299534 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:58.299802 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:58.299806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:58.299999 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:25:58.300266 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:58.300273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.301086 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.301087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:58.302286 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:58.302289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.302902 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.302903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:58.303051 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:58.303693 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:25:58.303717 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:25:58.305349 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:25:58.307523 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:58.307531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:58.307535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:58.307883 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:58.308118 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:58.308119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:58.308119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:58.308285 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:58.308628 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:25:58.308636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:25:58.308838 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 06:25:58.308965 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:25:58.309160 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":83,"threads_started":1}
I20260928 06:25:58.310240 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:58.298727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:45045 (local address 127.31.214.126:34407)
0928 06:25:58.298790 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:58.298792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:58.299109 (+ 317us) server_negotiation.cc:407] Connection header received
0928 06:25:58.299649 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:58.299651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:58.299663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:58.299720 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:58.300382 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.300990 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.302382 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.302744 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.302789 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:58.305460 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:58.305461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:58.305462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:58.305470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:58.307410 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:58.307977 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:58.307979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:58.307979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:58.308044 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:58.308369 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:58.308370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:58.308371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:58.308535 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:58.308599 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:58.309979 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0928 06:25:58.310092 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:25:58.310707 13248 heartbeater.cc:347] Connected to a master server at 127.31.214.126:34407
I20260928 06:25:58.310832 13248 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:58.311075 13248 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
W20260928 06:25:58.311971 13248 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:34407 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:25:58.312059 13248 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:58.312191 13248 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
I20260928 06:25:58.312687 13248 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:58.312777 13248 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
W20260928 06:25:58.313165 13248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:25:58.403152 13251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:25:58.403395 13251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:25:58.403437 13251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:25:58.404712 13251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:25:58.404778 13251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:25:58.404829 13251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:25:58.407529 13251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.214.84:60825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:41911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=45561
--tserver_master_addrs=127.31.214.126:34407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:25:58.407819 13251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:25:58.408151 13251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:25:58.413825 13256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:25:58.413858 13259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:58.414021 13251 server_base.cc:1034] running on GCE node
W20260928 06:25:58.414539 13257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:25:58.414821 13251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:25:58.415125 13251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:25:58.416294 13251 hybrid_clock.cc:648] HybridClock initialized: now 1790576758416270 us; error 42 us; skew 500 ppm
Sep 28 06:25:58 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:58.419657 13251 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:25:58.420259 13251 webserver.cc:460] Webserver started at http://127.31.214.67:45561/ using document root <none> and password file <none>
I20260928 06:25:58.420516 13251 fs_manager.cc:359] Metadata directory not provided
I20260928 06:25:58.420601 13251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:25:58 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
I20260928 06:25:58.438982 13251 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:25:58.440096 13267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:58.440326 13251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:25:58.440433 13251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "619106b547f14daa837f1e2569a2463a"
format_stamp: "Formatted at 2026-09-28 06:25:57 on dist-test-slave-q03f"
server_key: "cc7865cd4b7c2b3951aecca48d6a7b5e"
server_key_iv: "d8f8eb7fd0cb1d6c95c00523e79bdbcd"
server_key_version: "kuduclusterkey@0"
I20260928 06:25:58.440843 13251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:25:58.456002 13251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:25:58.458559 13251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:25:58.458736 13251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:25:58.459003 13251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:25:58.459398 13251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:25:58.459453 13251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:58.459522 13251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:25:58.459553 13251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:25:58.467279 13251 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:41911
I20260928 06:25:58.467330 13380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:41911 every 8 connection(s)
I20260928 06:25:58.467818 13251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:25:58.469969 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 13251
Sep 28 06:25:58 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576758, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:25:58.479079 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:58.469066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:53447 (local address 127.31.214.126:34407)
0928 06:25:58.469122 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:58.469124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:58.469135 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:25:58.469602 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:58.469603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:58.469617 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:58.469670 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:58.470526 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.471120 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.472290 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.472599 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.472692 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:58.475285 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:58.475287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:58.475288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:58.475296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:58.477276 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:58.477844 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:58.477846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:58.477846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:58.477938 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:58.478244 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:58.478246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:58.478246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:58.478385 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:58.478467 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:58.478889 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 06:25:58.478984 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:25:58.479277 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:58.468699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:34407 (local address 127.31.214.67:53447)
0928 06:25:58.468891 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:58.468939 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:58.469480 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:58.469751 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:58.469756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:58.470075 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:25:58.470396 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:58.470405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.471220 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.471221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:58.472188 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:58.472191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.472721 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.472722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:58.472851 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:58.473478 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:25:58.473492 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:25:58.475173 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:25:58.477388 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:58.477397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:58.477400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:58.477744 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:58.478031 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:25:58.478032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:25:58.478033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:25:58.478168 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:25:58.478501 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:25:58.478508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:25:58.478704 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 06:25:58.478838 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 06:25:58.479010 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":113,"threads_started":1}
I20260928 06:25:58.479837 13381 heartbeater.cc:347] Connected to a master server at 127.31.214.126:34407
I20260928 06:25:58.479952 13381 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:58.480154 13381 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
W20260928 06:25:58.480901 13381 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:34407 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:25:58.480988 13381 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:58.481066 13381 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
I20260928 06:25:58.481359 13381 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:58.481441 13381 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
W20260928 06:25:58.481760 13381 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:25:58.489493 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:58.470493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.31.214.126:34407)
0928 06:25:58.472344 (+ 1851us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:58.472346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:58.472559 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:25:58.472589 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:58.472591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:58.472604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:58.472674 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:58.481996 (+ 9322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.482752 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.483928 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:58.484151 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:58.484231 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:58.485611 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:58.485613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:58.485613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:58.485621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:58.487400 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:58.487797 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:58.487798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:58.487798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:58.487868 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:58.488864 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:58.488865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:58.488865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:58.489005 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:58.489209 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:58.489247 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:25:58.489344 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1822,"thread_start_us":57,"threads_started":1}
I20260928 06:25:59.148211 13114 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:59.148379 13114 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
I20260928 06:25:59.148919 12916 ts_manager.cc:194] Registered new tserver with Master: 0284c66eeea3454ca007a9e91c75f3b8 (127.31.214.65:46609)
I20260928 06:25:59.149930 12916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:33199
Sep 28 06:25:59 dist-test-slave-q03f krb5kdc[11920](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790576759, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:25:59.313431 13248 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:59.313611 13248 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
I20260928 06:25:59.314026 12916 ts_manager.cc:194] Registered new tserver with Master: 009db381c0aa4b83a7dfe1cb943bcc70 (127.31.214.66:36783)
I20260928 06:25:59.314568 12916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:45045
I20260928 06:25:59.481984 13381 heartbeater.cc:464] Registering TS with master...
I20260928 06:25:59.482174 13381 heartbeater.cc:511] Master 127.31.214.126:34407 requested a full tablet report, sending...
I20260928 06:25:59.482568 12916 ts_manager.cc:194] Registered new tserver with Master: 619106b547f14daa837f1e2569a2463a (127.31.214.67:41911)
I20260928 06:25:59.483186 12916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:53447
I20260928 06:25:59.484673 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:25:59.499346 13403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.485798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.31.214.126:34407)
0928 06:25:59.486131 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:59.486134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:59.489039 (+ 2905us) server_negotiation.cc:407] Connection header received
0928 06:25:59.489078 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:59.489080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:59.489096 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:59.489174 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:25:59.489984 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.490807 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.491985 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.492370 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.492461 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.494235 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:25:59.494236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:25:59.494237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:25:59.494246 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:25:59.497216 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:59.497664 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:59.497666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:59.497666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:59.497769 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:25:59.498040 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:25:59.498041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:25:59.498041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:25:59.498200 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:25:59.498301 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:59.498489 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:25:59.498586 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":94,"threads_started":1}
I20260928 06:25:59.501636 12916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:25:59.503940 12916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:25:59.517513 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.512206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46609 (local address 127.0.0.1:45540)
0928 06:25:59.512728 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:59.512772 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:59.512838 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:59.513303 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:59.513306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:59.513309 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:59.513392 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.513394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.514725 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.514727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.516109 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.516113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.517150 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.517152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.517283 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.517306 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 06:25:59.517361 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:25:59.517422 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":97,"threads_started":1}
I20260928 06:25:59.518308 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.512751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.31.214.65:46609)
0928 06:25:59.513090 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:59.513095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:59.513105 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:25:59.513141 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:59.513143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:59.513159 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:59.513469 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:59.513545 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.514596 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.516235 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.517024 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.518054 (+ 1030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.518142 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:59.518180 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:25:59.518225 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":66,"threads_started":1}
I20260928 06:25:59.519232 13042 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6 (DEFAULT_TABLE table=test-table [id=98108fc837c246b988fd452e12e359be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:25:59.519644 13042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:25:59.521142 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.512210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:36783 (local address 127.0.0.1:57536)
0928 06:25:59.512930 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:59.512938 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:59.512969 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:59.515500 (+ 2531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:59.515501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:59.515504 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:59.515555 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.515556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.518834 (+ 3278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.518835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.519882 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.519885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.520903 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.520904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.520974 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.520979 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:25:59.521013 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 06:25:59.521061 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":108,"threads_started":1}
I20260928 06:25:59.521514 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.512501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.31.214.66:36783)
0928 06:25:59.514959 (+ 2458us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:59.514963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:59.514974 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:25:59.515012 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:59.515014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:59.515033 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:59.515635 (+ 602us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:59.515725 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.516913 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.519996 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.520779 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.521282 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.521364 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:59.521394 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:25:59.521438 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2421,"thread_start_us":52,"threads_started":1}
I20260928 06:25:59.522140 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.512460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:41911 (local address 127.0.0.1:55168)
0928 06:25:59.517064 (+ 4604us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:59.517076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:59.517129 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:59.517762 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:59.517763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:59.517766 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:59.517809 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.517811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.519188 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.519189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.520685 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.520688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.521881 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.521882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.521966 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.521971 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:25:59.522022 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:25:59.522069 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4580,"thread_start_us":63,"threads_started":1}
I20260928 06:25:59.522270 13169 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6 (DEFAULT_TABLE table=test-table [id=98108fc837c246b988fd452e12e359be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:25:59.522593 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.513037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.31.214.67:41911)
0928 06:25:59.514870 (+ 1833us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:59.514876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:59.517193 (+ 2317us) server_negotiation.cc:407] Connection header received
0928 06:25:59.517238 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:59.517241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:59.517258 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:59.517330 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:59.517913 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.519062 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.520781 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.521693 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.521770 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.521828 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:59.522413 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 06:25:59.522516 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1773,"thread_start_us":78,"threads_started":1}
I20260928 06:25:59.522672 13169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:25:59.523617 13305 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6 (DEFAULT_TABLE table=test-table [id=98108fc837c246b988fd452e12e359be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:25:59.523988 13305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:25:59.528610 13418 tablet_bootstrap.cc:492] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: Bootstrap starting.
I20260928 06:25:59.528924 13417 tablet_bootstrap.cc:492] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70: Bootstrap starting.
I20260928 06:25:59.529385 13418 tablet_bootstrap.cc:654] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: Neither blocks nor log segments found. Creating new log.
I20260928 06:25:59.529721 13418 log.cc:824] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:25:59.530130 13416 tablet_bootstrap.cc:492] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8: Bootstrap starting.
I20260928 06:25:59.530829 13417 tablet_bootstrap.cc:654] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70: Neither blocks nor log segments found. Creating new log.
I20260928 06:25:59.530829 13416 tablet_bootstrap.cc:654] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8: Neither blocks nor log segments found. Creating new log.
I20260928 06:25:59.531239 13417 log.cc:824] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70: Log is configured to *not* fsync() on all Append() calls
I20260928 06:25:59.531638 13416 log.cc:824] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:25:59.532313 13418 tablet_bootstrap.cc:492] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: No bootstrap required, opened a new log
I20260928 06:25:59.532439 13418 ts_tablet_manager.cc:1397] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 06:25:59.532512 13417 tablet_bootstrap.cc:492] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70: No bootstrap required, opened a new log
I20260928 06:25:59.532645 13417 ts_tablet_manager.cc:1397] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 06:25:59.534834 13418 raft_consensus.cc:348] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } }
I20260928 06:25:59.535044 13418 raft_consensus.cc:374] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:25:59.535092 13418 raft_consensus.cc:729] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619106b547f14daa837f1e2569a2463a, State: Initialized, Role: FOLLOWER
I20260928 06:25:59.535243 13418 consensus_queue.cc:260] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } }
I20260928 06:25:59.535576 13418 ts_tablet_manager.cc:1428] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:25:59.535718 13381 heartbeater.cc:503] Master 127.31.214.126:34407 was elected leader, sending a full tablet report...
I20260928 06:25:59.538160 13416 tablet_bootstrap.cc:492] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8: No bootstrap required, opened a new log
I20260928 06:25:59.538239 13416 ts_tablet_manager.cc:1397] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 06:25:59.541227 13416 raft_consensus.cc:348] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } }
I20260928 06:25:59.541421 13416 raft_consensus.cc:374] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:25:59.541460 13416 raft_consensus.cc:729] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0284c66eeea3454ca007a9e91c75f3b8, State: Initialized, Role: FOLLOWER
I20260928 06:25:59.541589 13416 consensus_queue.cc:260] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } }
I20260928 06:25:59.541996 13416 ts_tablet_manager.cc:1428] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:25:59.542203 13114 heartbeater.cc:503] Master 127.31.214.126:34407 was elected leader, sending a full tablet report...
I20260928 06:25:59.544265 13417 raft_consensus.cc:348] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } }
I20260928 06:25:59.544533 13417 raft_consensus.cc:374] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:25:59.544605 13417 raft_consensus.cc:729] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009db381c0aa4b83a7dfe1cb943bcc70, State: Initialized, Role: FOLLOWER
I20260928 06:25:59.544842 13417 consensus_queue.cc:260] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } }
I20260928 06:25:59.545189 13248 heartbeater.cc:503] Master 127.31.214.126:34407 was elected leader, sending a full tablet report...
I20260928 06:25:59.545179 13417 ts_tablet_manager.cc:1428] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260928 06:25:59.548583 13249 tablet.cc:2369] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:25:59 dist-test-slave-q03f krb5kdc[11920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576759, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for HTTP/127.31.214.84@KRBTEST.COM
W20260928 06:25:59.626801 13115 tablet.cc:2369] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:25:59.678792 13424 raft_consensus.cc:482] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:25:59.678968 13424 raft_consensus.cc:504] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } }
I20260928 06:25:59.679574 13424 leader_election.cc:288] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0284c66eeea3454ca007a9e91c75f3b8 (127.31.214.65:46609), 619106b547f14daa837f1e2569a2463a (127.31.214.67:41911)
I20260928 06:25:59.683732 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.679898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46609 (local address 127.31.214.66:37979)
0928 06:25:59.680294 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:59.680306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:59.680394 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:59.680575 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:59.680576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:59.680578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:59.680656 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.680658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.681519 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.681520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.682687 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.682690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.683470 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.683472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.683534 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.683538 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:25:59.683589 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:25:59.683628 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":116,"threads_started":1}
I20260928 06:25:59.683772 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.680145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:37979 (local address 127.31.214.65:46609)
0928 06:25:59.680189 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:59.680192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:59.680365 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:25:59.680469 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:59.680470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:59.680479 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:59.680508 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:59.680725 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.681419 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.682797 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.683332 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.683382 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.683392 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:59.683633 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:25:59.683696 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:25:59.683995 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.680206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:47461 (local address 127.31.214.67:41911)
0928 06:25:59.680249 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:59.680251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:59.680373 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:25:59.680470 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:59.680471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:59.680480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:59.680508 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:25:59.680935 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.681590 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.683171 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.683551 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.683650 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.683659 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:59.683854 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:25:59.683905 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:25:59.684231 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.679899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:41911 (local address 127.31.214.66:47461)
0928 06:25:59.680312 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:59.680322 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:25:59.680398 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:25:59.680575 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:25:59.680576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:25:59.680580 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:25:59.680833 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.680836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.681713 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.681714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.683059 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:25:59.683062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.683692 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.683694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:25:59.683763 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.683767 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:25:59.684073 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 06:25:59.684123 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":4,"spinlock_wait_cycles":12032,"thread_start_us":191,"threads_started":1}
I20260928 06:25:59.684294 13068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7896a9203c4cf5bc8b3b3b4e8bfda6" candidate_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0284c66eeea3454ca007a9e91c75f3b8" is_pre_election: true
I20260928 06:25:59.684489 13068 raft_consensus.cc:2454] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009db381c0aa4b83a7dfe1cb943bcc70 in term 0.
I20260928 06:25:59.684626 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7896a9203c4cf5bc8b3b3b4e8bfda6" candidate_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619106b547f14daa837f1e2569a2463a" is_pre_election: true
I20260928 06:25:59.684770 13137 leader_election.cc:302] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009db381c0aa4b83a7dfe1cb943bcc70, 0284c66eeea3454ca007a9e91c75f3b8; no voters:
I20260928 06:25:59.684821 13335 raft_consensus.cc:2454] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009db381c0aa4b83a7dfe1cb943bcc70 in term 0.
I20260928 06:25:59.684916 13424 raft_consensus.cc:2781] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:25:59.684966 13424 raft_consensus.cc:482] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:25:59.684998 13424 raft_consensus.cc:3037] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:25:59.686277 13424 raft_consensus.cc:504] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } }
I20260928 06:25:59.686429 13424 leader_election.cc:288] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [CANDIDATE]: Term 1 election: Requested vote from peers 0284c66eeea3454ca007a9e91c75f3b8 (127.31.214.65:46609), 619106b547f14daa837f1e2569a2463a (127.31.214.67:41911)
I20260928 06:25:59.686642 13068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7896a9203c4cf5bc8b3b3b4e8bfda6" candidate_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0284c66eeea3454ca007a9e91c75f3b8"
I20260928 06:25:59.686653 13335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7896a9203c4cf5bc8b3b3b4e8bfda6" candidate_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619106b547f14daa837f1e2569a2463a"
I20260928 06:25:59.686730 13335 raft_consensus.cc:3037] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:25:59.686750 13068 raft_consensus.cc:3037] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:25:59.687567 13068 raft_consensus.cc:2454] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009db381c0aa4b83a7dfe1cb943bcc70 in term 1.
I20260928 06:25:59.687754 13137 leader_election.cc:302] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009db381c0aa4b83a7dfe1cb943bcc70, 0284c66eeea3454ca007a9e91c75f3b8; no voters:
I20260928 06:25:59.687809 13335 raft_consensus.cc:2454] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009db381c0aa4b83a7dfe1cb943bcc70 in term 1.
I20260928 06:25:59.687983 13424 raft_consensus.cc:2781] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:25:59.688217 13424 raft_consensus.cc:686] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 1 LEADER]: Becoming Leader. State: Replica: 009db381c0aa4b83a7dfe1cb943bcc70, State: Running, Role: LEADER
I20260928 06:25:59.688349 13424 consensus_queue.cc:237] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } }
I20260928 06:25:59.689327 12916 catalog_manager.cc:5495] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009db381c0aa4b83a7dfe1cb943bcc70 (127.31.214.66). New cstate: current_term: 1 leader_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009db381c0aa4b83a7dfe1cb943bcc70" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36783 } health_report { overall_health: HEALTHY } } }
I20260928 06:25:59.701193 13413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.698604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.31.214.66:36783)
0928 06:25:59.698668 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:25:59.698670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:25:59.698696 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:25:59.698783 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:25:59.698784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:25:59.698813 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:25:59.698895 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:25:59.699051 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.699555 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.700355 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:25:59.700558 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:25:59.700636 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:25:59.700826 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:25:59.700941 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:25:59.701040 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:25:59.701059 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:25:59.701083 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:25:59.703420 13068 raft_consensus.cc:1261] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 0284c66eeea3454ca007a9e91c75f3b8 [term 1 FOLLOWER]: Refusing update from remote peer 009db381c0aa4b83a7dfe1cb943bcc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:25:59.703420 13335 raft_consensus.cc:1261] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [term 1 FOLLOWER]: Refusing update from remote peer 009db381c0aa4b83a7dfe1cb943bcc70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:25:59.703796 13424 consensus_queue.cc:1035] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619106b547f14daa837f1e2569a2463a" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:25:59.704095 13424 consensus_queue.cc:1035] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0284c66eeea3454ca007a9e91c75f3b8" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:25:59.708418 13431 mvcc.cc:205] Tried to move back new op lower bound from 7334202407742033920 to 7334202407684083712. Current Snapshot: MvccSnapshot[applied={T|T < 7334202407742033920}]
I20260928 06:25:59.708608 13432 mvcc.cc:205] Tried to move back new op lower bound from 7334202407742033920 to 7334202407684083712. Current Snapshot: MvccSnapshot[applied={T|T < 7334202407742033920}]
I20260928 06:25:59.710145 12916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:25:59.710253 12916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:25:59.711059 12916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a8b7739704d4563bc2e56771b84ae5e: Sending DeleteTablet for 3 replicas of tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6
I20260928 06:25:59.711524 13305 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:25:59 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:55168
I20260928 06:25:59.711524 13042 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:25:59 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:45540
I20260928 06:25:59.712263 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 12954
I20260928 06:25:59.712270 13169 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:25:59 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:57536
I20260928 06:25:59.718284 13441 tablet_replica.cc:330] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70: stopping tablet replica
I20260928 06:25:59.718446 13441 raft_consensus.cc:2229] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:25:59.718618 13441 raft_consensus.cc:2258] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 009db381c0aa4b83a7dfe1cb943bcc70 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 06:25:59.723418 12902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:46609 (error 108)
I20260928 06:25:59.723827 13440 tablet_replica.cc:330] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: stopping tablet replica
I20260928 06:25:59.723968 13440 raft_consensus.cc:2229] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:25:59.724097 13440 raft_consensus.cc:2258] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:25:59.724552 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 13116
I20260928 06:25:59.725263 13440 ts_tablet_manager.cc:1905] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:25:59.727497 13440 ts_tablet_manager.cc:1918] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:25:59.727566 13440 log.cc:1196] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/8f7896a9203c4cf5bc8b3b3b4e8bfda6
I20260928 06:25:59.727839 13440 ts_tablet_manager.cc:1939] T 8f7896a9203c4cf5bc8b3b3b4e8bfda6 P 619106b547f14daa837f1e2569a2463a: Deleting consensus metadata
I20260928 06:25:59.728579 12903 catalog_manager.cc:4841] TS 619106b547f14daa837f1e2569a2463a (127.31.214.67:41911): tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6 (table test-table [id=98108fc837c246b988fd452e12e359be]) successfully deleted
I20260928 06:25:59.731817 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 13251
I20260928 06:25:59.733860 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.733699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46609 (local address 127.0.0.1:45544)
0928 06:25:59.733764 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:59.733810 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:25:59.734180 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:25:59.734088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:36783 (local address 127.0.0.1:57558)
0928 06:25:59.734128 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:25:59.734150 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.66:36783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 06:25:59.734324 12902 catalog_manager.cc:4584] TS 0284c66eeea3454ca007a9e91c75f3b8 (127.31.214.65:46609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f7896a9203c4cf5bc8b3b3b4e8bfda6: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46609: connect: Connection refused (error 111)
I20260928 06:25:59.737740 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 12883
I20260928 06:25:59.744369 32601 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:26:00.079563 32601 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:26:00.079679 32601 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:26:00.424863 32601 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:26:00.425 UTC [11950] LOG: received smart shutdown request
2026-09-28 06:26:00.429 UTC [11950] LOG: background worker "logical replication launcher" (PID 11959) exited with exit code 1
2026-09-28 06:26:00.429 UTC [11954] LOG: shutting down
2026-09-28 06:26:00.657 UTC [11950] LOG: database system is shut down
2026-09-28T06:26:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52712 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:26:00 dist-test-slave-q03f krb5kdc[13451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:26:00 dist-test-slave-q03f krb5kdc[13451](info): set up 2 sockets
Sep 28 06:26:00 dist-test-slave-q03f krb5kdc[13451](info): commencing operation
krb5kdc: starting...
W20260928 06:26:01.874408 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:26:01 dist-test-slave-q03f krb5kdc[13451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:26:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:26:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:26:02.009588 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:36387
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:36387,127.31.214.125:37421,127.31.214.124:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:02.107014 13464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:02.107260 13464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:02.107287 13464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:02.108779 13464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:26:02.108851 13464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:02.108866 13464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:02.108879 13464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:26:02.108892 13464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:26:02.110800 13464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:36387,127.31.214.125:37421,127.31.214.124:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:02.111119 13464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:02.111519 13464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:02.117108 13470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:02.117108 13469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:02.117188 13472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:02.117409 13464 server_base.cc:1034] running on GCE node
I20260928 06:26:02.117790 13464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:02.118185 13464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:02.119387 13464 hybrid_clock.cc:648] HybridClock initialized: now 1790576762119363 us; error 43 us; skew 500 ppm
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:02.122246 13464 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:26:02.122805 13464 webserver.cc:460] Webserver started at http://127.31.214.126:40685/ using document root <none> and password file <none>
I20260928 06:26:02.123106 13464 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:02.123179 13464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:02.123350 13464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:02.124643 13464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "b58d47cf40c8418d98fbc5b45ea5c538"
format_stamp: "Formatted at 2026-09-28 06:26:02 on dist-test-slave-q03f"
server_key: "9ff9a0aa8aab0d2d4d77f3ccb64d9986"
server_key_iv: "b4804e3fce7ba18bf47353db7a02e2ef"
server_key_version: "encryptionkey@0"
I20260928 06:26:02.125098 13464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "b58d47cf40c8418d98fbc5b45ea5c538"
format_stamp: "Formatted at 2026-09-28 06:26:02 on dist-test-slave-q03f"
server_key: "9ff9a0aa8aab0d2d4d77f3ccb64d9986"
server_key_iv: "b4804e3fce7ba18bf47353db7a02e2ef"
server_key_version: "encryptionkey@0"
I20260928 06:26:02.126885 13464 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 06:26:02.127779 13479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:02.128000 13464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:26:02.128129 13464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "b58d47cf40c8418d98fbc5b45ea5c538"
format_stamp: "Formatted at 2026-09-28 06:26:02 on dist-test-slave-q03f"
server_key: "9ff9a0aa8aab0d2d4d77f3ccb64d9986"
server_key_iv: "b4804e3fce7ba18bf47353db7a02e2ef"
server_key_version: "encryptionkey@0"
I20260928 06:26:02.128221 13464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:02.139371 13464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:02.142086 13464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:02.142267 13464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:02.146826 13464 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:36387
I20260928 06:26:02.146896 13531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:36387 every 8 connection(s)
I20260928 06:26:02.147285 13464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:26:02.148231 13532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:02.149886 13532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:26:02.155701 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 13464
I20260928 06:26:02.155848 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:26:02.156088 32601 external_mini_cluster.cc:1428] Setting key b5d38a80a0812707675dd9e69c67b3ac
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:02.161938 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.150819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:36387 (local address 127.0.0.1:43544)
0928 06:26:02.151082 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.151111 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:02.151648 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:02.151982 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:02.151984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:02.152346 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:02.152517 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.152522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.153646 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.153647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.154539 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.154542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.155061 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.155063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.155136 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.155923 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:02.155935 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:02.157827 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:02.160031 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.160038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.160041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.160296 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.160593 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.160594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.160594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.160729 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.161375 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:02.161382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:02.161451 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 06:26:02.161535 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:26:02.161724 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":115,"threads_started":1}
I20260928 06:26:02.162134 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.150819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.31.214.126:36387)
0928 06:26:02.151024 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.151032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.151217 (+ 185us) server_negotiation.cc:407] Connection header received
0928 06:26:02.151832 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.151835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.151855 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.151950 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.152633 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.153526 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.154630 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.154974 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.155065 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.157980 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.157989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.157989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.158005 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.159891 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.160392 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.160395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.160396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.160487 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.160811 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.160812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.160812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.161281 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.161392 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.161600 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.161849 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":158,"thread_start_us":112,"threads_started":1}
I20260928 06:26:02.163033 13532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:26:02.163601 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.163319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:37421 (local address 127.0.0.1:45820)
0928 06:26:02.163523 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.163578 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260928 06:26:02.163810 13480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
I20260928 06:26:02.164501 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.164388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:37421 (local address 127.0.0.1:45836)
0928 06:26:02.164450 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.164481 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 06:26:02.164672 13532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.125:37421: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.31.214.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab.
Entry for principal kudu/127.31.214.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab.
I20260928 06:26:02.190574 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.125:37421
--webserver_interface=127.31.214.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab
--principal=kudu/127.31.214.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:36387,127.31.214.125:37421,127.31.214.124:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:02.214039 13532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } attempt: 1
I20260928 06:26:02.214582 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.214432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:37421 (local address 127.0.0.1:45852)
0928 06:26:02.214505 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.214549 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:26:02.214977 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.214872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:37421 (local address 127.0.0.1:45858)
0928 06:26:02.214924 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.214952 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:26:02.215070 13532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.125:37421: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
W20260928 06:26:02.276860 13539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:02.277123 13539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:02.277150 13539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:02.278489 13539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:26:02.278542 13539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:02.278558 13539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:02.278570 13539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:26:02.278584 13539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:26:02.280262 13539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:36387,127.31.214.125:37421,127.31.214.124:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab
--principal=kudu/127.31.214.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.125:37421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.214.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:02.280550 13539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:02.280825 13539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:26:02.283223 13532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } attempt: 2
I20260928 06:26:02.283735 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.283565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:37421 (local address 127.0.0.1:45874)
0928 06:26:02.283634 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.283688 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:26:02.284135 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.284037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:37421 (local address 127.0.0.1:45888)
0928 06:26:02.284085 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.284114 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:26:02.284247 13532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.125:37421: Network error: Client connection negotiation failed: client connection to 127.31.214.125:37421: connect: Connection refused (error 111)
W20260928 06:26:02.286571 13545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:02.286571 13547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:02.286593 13544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:02.286959 13539 server_base.cc:1034] running on GCE node
I20260928 06:26:02.287240 13539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:02.287601 13539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:02.288780 13539 hybrid_clock.cc:648] HybridClock initialized: now 1790576762288747 us; error 47 us; skew 500 ppm
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:02.291816 13539 init.cc:377] Logged in from keytab as kudu/127.31.214.125@KRBTEST.COM (short username kudu)
I20260928 06:26:02.292484 13539 webserver.cc:460] Webserver started at http://127.31.214.125:45011/ using document root <none> and password file <none>
I20260928 06:26:02.292791 13539 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:02.292876 13539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:02.293048 13539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:02.294394 13539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/data/instance:
uuid: "48b1923294684160ba9da7619c3d2af7"
format_stamp: "Formatted at 2026-09-28 06:26:02 on dist-test-slave-q03f"
server_key: "1f92b9917070b0ad20fb3e71eaa2c7af"
server_key_iv: "9ddd1ae363a133887d2071e5cee8f70c"
server_key_version: "encryptionkey@0"
I20260928 06:26:02.294811 13539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/wal/instance:
uuid: "48b1923294684160ba9da7619c3d2af7"
format_stamp: "Formatted at 2026-09-28 06:26:02 on dist-test-slave-q03f"
server_key: "1f92b9917070b0ad20fb3e71eaa2c7af"
server_key_iv: "9ddd1ae363a133887d2071e5cee8f70c"
server_key_version: "encryptionkey@0"
I20260928 06:26:02.296437 13539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:26:02.297420 13554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:02.297688 13539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:02.297814 13539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/wal
uuid: "48b1923294684160ba9da7619c3d2af7"
format_stamp: "Formatted at 2026-09-28 06:26:02 on dist-test-slave-q03f"
server_key: "1f92b9917070b0ad20fb3e71eaa2c7af"
server_key_iv: "9ddd1ae363a133887d2071e5cee8f70c"
server_key_version: "encryptionkey@0"
I20260928 06:26:02.297912 13539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:02.309928 13539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:02.312559 13539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:02.312745 13539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:02.317428 13539 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.125:37421
I20260928 06:26:02.317515 13606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.125:37421 every 8 connection(s)
I20260928 06:26:02.317902 13539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
I20260928 06:26:02.318925 13607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:02.320663 13607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:26:02.326288 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 13539
I20260928 06:26:02.326421 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-1/wal/instance
I20260928 06:26:02.326632 32601 external_mini_cluster.cc:1428] Setting key 35b893bb5a5a9a870ad1145bc088ed85
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:02.331456 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.321554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.31.214.126:36387)
0928 06:26:02.321672 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.321674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.321855 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:26:02.322401 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.322403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.322413 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.322492 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.323143 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.323719 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.324870 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.325112 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.325174 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.328004 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.328006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.328007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.328015 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.329740 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.330265 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.330267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.330267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.330349 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.330644 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.330646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.330651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.330784 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.330867 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.331261 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.331357 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 06:26:02.331693 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.321514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:36387 (local address 127.0.0.1:43558)
0928 06:26:02.321698 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.321745 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:02.322291 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:02.322538 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:02.322541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:02.322809 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:02.323015 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.323019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.323826 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.323827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.324747 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.324750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.325234 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.325236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.325427 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.326144 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:02.326154 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:02.327846 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:02.329840 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.329848 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.329850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.330185 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.330420 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.330421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.330421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.330540 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.330870 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:02.330879 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:02.331023 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 06:26:02.331218 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 06:26:02.331400 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260928 06:26:02.332453 13607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.214.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.124@KRBTEST.COM" created.
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:26:02.342154 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.332810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:37421 (local address 127.0.0.1:45892)
0928 06:26:02.332867 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.332879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:02.332925 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:02.333258 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:02.333259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:02.333377 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:02.333433 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.333434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.334893 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.334895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.335544 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.335546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.336063 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.336065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.336131 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.336626 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:02.336633 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:02.339144 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:02.340990 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.340991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.340992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.341167 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.341382 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.341382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.341383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.341438 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.341751 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:02.341752 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:02.341792 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:26:02.341859 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:26:02.342028 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:26:02.342197 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.332876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.31.214.125:37421)
0928 06:26:02.333070 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.333076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.333102 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:26:02.333125 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.333128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.333143 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.333228 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.333514 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.334777 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.335630 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.335964 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.336029 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.339243 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.339253 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.339254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.339261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.340885 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.341252 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.341256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.341256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.341315 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.341519 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.341519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.341520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.341674 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.341762 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.341900 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.342088 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260928 06:26:02.342895 13607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:26:02.343341 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.343182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:51962)
0928 06:26:02.343236 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.343305 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:26:02.343541 13556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
I20260928 06:26:02.344063 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.343946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:51968)
0928 06:26:02.343996 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.344031 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:26:02.344218 13607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:41243: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab.
Entry for principal kudu/127.31.214.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab.
I20260928 06:26:02.358134 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.124:41243
--webserver_interface=127.31.214.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab
--principal=kudu/127.31.214.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:36387,127.31.214.125:37421,127.31.214.124:41243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:02.375423 13532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } attempt: 3
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:26:02.383615 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.376010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.31.214.125:37421)
0928 06:26:02.376174 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.376176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.376184 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:26:02.376210 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.376212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.376224 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.376298 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.376716 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.377335 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.378227 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.378472 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.378573 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.380821 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.380823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.380824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.380834 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.382361 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.382801 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.382803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.382803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.382864 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.383088 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.383089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.383089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.383212 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.383270 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.383465 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.383543 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:26:02.383643 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.375876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:37421 (local address 127.0.0.1:45902)
0928 06:26:02.375977 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.375997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:02.376077 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:02.376308 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:02.376309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:02.376489 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:02.376578 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.376580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.377441 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.377443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.378118 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.378121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.378580 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.378581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.378676 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.379203 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:02.379208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:02.380720 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:02.382448 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.382449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.382450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.382703 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.382943 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.382944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.382944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.383018 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.383276 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:02.383277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:02.383338 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 06:26:02.383414 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:26:02.383540 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 06:26:02.384745 13532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:26:02.385154 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.385032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:51978)
0928 06:26:02.385088 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.385125 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:26:02.385538 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.385440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:51986)
0928 06:26:02.385488 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.385515 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:26:02.385653 13532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:41243: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
I20260928 06:26:02.393477 13607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } attempt: 1
I20260928 06:26:02.394172 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.394007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52002)
0928 06:26:02.394077 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.394130 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:26:02.394629 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.394497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52014)
0928 06:26:02.394555 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.394594 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:26:02.394747 13607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:41243: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
I20260928 06:26:02.416815 13532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } attempt: 1
I20260928 06:26:02.417438 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.417259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52030)
0928 06:26:02.417353 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.417402 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 06:26:02.417856 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.417766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52044)
0928 06:26:02.417807 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.417834 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:26:02.417996 13532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:41243: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
W20260928 06:26:02.447630 13614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:02.447872 13614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:02.447896 13614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:02.449057 13614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:26:02.449103 13614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:02.449118 13614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:02.449133 13614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:26:02.449146 13614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:26:02.450747 13614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:36387,127.31.214.125:37421,127.31.214.124:41243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab
--principal=kudu/127.31.214.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.124:41243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.214.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:02.450964 13614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:02.451160 13614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:02.456835 13619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:02.456861 13620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:02.456842 13622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:02.457196 13614 server_base.cc:1034] running on GCE node
I20260928 06:26:02.457517 13614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:02.457849 13614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:02.459028 13614 hybrid_clock.cc:648] HybridClock initialized: now 1790576762459011 us; error 34 us; skew 500 ppm
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:02.462038 13614 init.cc:377] Logged in from keytab as kudu/127.31.214.124@KRBTEST.COM (short username kudu)
I20260928 06:26:02.462643 13614 webserver.cc:460] Webserver started at http://127.31.214.124:42803/ using document root <none> and password file <none>
I20260928 06:26:02.462920 13614 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:02.462886 13607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } attempt: 2
I20260928 06:26:02.462998 13614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:02.463186 13614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:02.463392 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.463191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52048)
0928 06:26:02.463320 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.463361 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 06:26:02.463768 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.463658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52056)
0928 06:26:02.463707 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.463739 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:26:02.463855 13607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:41243: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
I20260928 06:26:02.464478 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/data/instance:
uuid: "318d98ffb35349b0979d62e24c6c5f04"
format_stamp: "Formatted at 2026-09-28 06:26:02 on dist-test-slave-q03f"
server_key: "97bbcf2592adbf6faaea2789136cb32f"
server_key_iv: "a32a16045cba01ec23bd5986e2b8b76b"
server_key_version: "encryptionkey@0"
I20260928 06:26:02.464860 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/wal/instance:
uuid: "318d98ffb35349b0979d62e24c6c5f04"
format_stamp: "Formatted at 2026-09-28 06:26:02 on dist-test-slave-q03f"
server_key: "97bbcf2592adbf6faaea2789136cb32f"
server_key_iv: "a32a16045cba01ec23bd5986e2b8b76b"
server_key_version: "encryptionkey@0"
I20260928 06:26:02.466408 13614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:26:02.467365 13629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:02.467607 13614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:26:02.467728 13614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/wal
uuid: "318d98ffb35349b0979d62e24c6c5f04"
format_stamp: "Formatted at 2026-09-28 06:26:02 on dist-test-slave-q03f"
server_key: "97bbcf2592adbf6faaea2789136cb32f"
server_key_iv: "a32a16045cba01ec23bd5986e2b8b76b"
server_key_version: "encryptionkey@0"
I20260928 06:26:02.467808 13614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:02.493242 13532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } attempt: 2
I20260928 06:26:02.493849 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.493674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52070)
0928 06:26:02.493756 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.493802 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:26:02.494262 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.494150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52078)
0928 06:26:02.494207 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.494241 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:26:02.494352 13532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:41243: Network error: Client connection negotiation failed: client connection to 127.31.214.124:41243: connect: Connection refused (error 111)
I20260928 06:26:02.522460 13614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:02.525123 13614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:02.525314 13614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:02.529834 13614 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.124:41243
I20260928 06:26:02.529878 13681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.124:41243 every 8 connection(s)
I20260928 06:26:02.530333 13614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
I20260928 06:26:02.531347 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:02.533046 13682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:26:02.534543 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 13614
I20260928 06:26:02.534643 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790576570121568-32601-0/minicluster-data/master-2/wal/instance
I20260928 06:26:02.534860 32601 external_mini_cluster.cc:1428] Setting key bd91e50fb887954580c00da339469905
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
I20260928 06:26:02.544539 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.533987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.31.214.126:36387)
0928 06:26:02.534046 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.534048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.534180 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:26:02.534771 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.534773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.534787 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.534843 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.535699 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.536298 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.537565 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.537867 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.537931 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.540606 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.540607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.540608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.540617 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.542679 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.543283 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.543284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.543285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.543438 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.543758 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.543759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.543759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.543915 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.543995 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.544346 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.544451 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:26:02.544889 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.533818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:36387 (local address 127.0.0.1:43570)
0928 06:26:02.534045 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.534099 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:02.534644 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:02.534925 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:02.534929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:02.535301 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:02.535595 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.535602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.536428 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.536429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.537466 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.537470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.537998 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.537999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.538169 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.538785 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:02.538797 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:02.540505 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:02.542774 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.542801 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.542804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.543187 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.543535 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.543535 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.543536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.543668 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.544011 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:02.544018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:02.544160 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 06:26:02.544309 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 06:26:02.544550 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
I20260928 06:26:02.545531 13682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:26:02.545696 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.535322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.31.214.124:41243)
0928 06:26:02.535517 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.535521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.535583 (+ 62us) server_negotiation.cc:407] Connection header received
0928 06:26:02.535710 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.535714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.535739 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.535845 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.536745 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.537653 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.538790 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.539088 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.539149 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.542142 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.542154 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.542154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.542163 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.544202 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.544675 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.544678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.544678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.544735 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.544982 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.544983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.544983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.545188 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.545440 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.545491 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.545573 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":168,"thread_start_us":99,"threads_started":1}
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:02.553846 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.546004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.31.214.125:37421)
0928 06:26:02.546080 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.546082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.546093 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:26:02.546120 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.546121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.546132 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.546200 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.546525 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.547303 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.548302 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.548607 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.548687 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.550748 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.550749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.550749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.550757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.552586 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.553019 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.553021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.553022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.553085 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.553295 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.553296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.553297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.553451 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.553525 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.553660 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.553747 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:26:02.553879 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.545858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:37421 (local address 127.0.0.1:45914)
0928 06:26:02.545897 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.545911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:02.545984 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:02.546231 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:02.546233 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:02.546391 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:02.546447 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.546449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.547411 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.547412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.548206 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.548208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.548689 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.548690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.548750 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.549222 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:02.549227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:02.550653 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:02.552710 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.552711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.552712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.552935 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.553160 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.553161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.553161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.553224 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.553519 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:02.553520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:02.553559 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:26:02.553610 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:26:02.553765 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 06:26:02.554421 13682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:26:02.554934 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.546603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.31.214.126:36387)
0928 06:26:02.546638 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.546640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.546647 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:26:02.546706 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.546706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.546713 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.546753 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.547305 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.548067 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.549100 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.549425 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.549508 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.551906 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.551907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.551908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.551916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.553629 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.554077 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.554078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.554079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.554160 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.554381 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.554383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.554383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.554515 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.554721 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.554758 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.554834 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:26:02.555009 13607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } attempt: 3
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:26:02.563582 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.555680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52114)
0928 06:26:02.555750 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.555762 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:02.555811 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:02.556100 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:02.556101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:02.556258 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:02.556300 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.556302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.556959 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.556960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.557695 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.557696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.558289 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.558291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.558349 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.558752 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:02.558757 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:02.560247 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:02.562363 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.562364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.562365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.562546 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.562755 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.562755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.562756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.562820 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.563232 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:02.563234 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:02.563284 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:26:02.563333 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:26:02.563449 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:26:02.564090 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.555785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.31.214.124:41243)
0928 06:26:02.555966 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.555968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.555975 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:26:02.555994 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.555994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.556005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.556078 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.556397 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.556877 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.557800 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.558187 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.558285 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.560378 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.560379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.560379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.560386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.562275 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.562622 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.562623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.562623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.562682 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.562884 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.562884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.562885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.563139 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.563801 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.563841 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.563929 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":129,"threads_started":1}
I20260928 06:26:02.564446 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.554759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52098)
0928 06:26:02.554791 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.554801 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:02.554839 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:02.555202 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:02.555204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:02.555339 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:02.555373 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.555374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.556266 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.556267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.557049 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.557052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.557518 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.557519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.557581 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.558059 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:02.558064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:02.559513 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:02.562981 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.562982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.562983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.563299 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.563524 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.563525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.563525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.563610 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.564136 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:02.564137 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:02.564188 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:26:02.564251 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:26:02.564350 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 06:26:02.564800 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.554794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.31.214.124:41243)
0928 06:26:02.555075 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.555077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.555085 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:26:02.555104 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.555105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.555115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.555173 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.555449 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.556179 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.557128 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.557437 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.557505 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.559599 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.559600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.559601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.559608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.562910 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.563381 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.563382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.563383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.563441 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.563951 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.563951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.563952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.564077 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.564367 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.564410 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.564697 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260928 06:26:02.565028 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.556148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.31.214.125:37421)
0928 06:26:02.556193 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.556195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.556202 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:26:02.556267 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.556267 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.556276 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.556303 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.556951 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.557509 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.558534 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.558811 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.558903 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.562143 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.562144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.562145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.562152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.563758 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.564081 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.564082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.564083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.564138 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.564364 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.564364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.564365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.564485 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.564657 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.564850 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.564924 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":20}
I20260928 06:26:02.566351 13607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7: Bootstrap starting.
I20260928 06:26:02.567166 13607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:02.567512 13682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04: Bootstrap starting.
I20260928 06:26:02.567519 13607 log.cc:824] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:02.568259 13682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:02.568454 13607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7: No bootstrap required, opened a new log
I20260928 06:26:02.568584 13682 log.cc:824] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:02.569458 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:26:02.569514 13682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04: No bootstrap required, opened a new log
I20260928 06:26:02.570830 13607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } }
I20260928 06:26:02.571002 13607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:02.571049 13607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b1923294684160ba9da7619c3d2af7, State: Initialized, Role: FOLLOWER
I20260928 06:26:02.571173 13607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } }
I20260928 06:26:02.571543 13607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:26:02.571498 13695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.571659 13695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:26:02.571811 13682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } }
I20260928 06:26:02.571959 13682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:02.572137 13682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318d98ffb35349b0979d62e24c6c5f04, State: Initialized, Role: FOLLOWER
I20260928 06:26:02.572355 13682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } }
I20260928 06:26:02.572664 13702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.572710 13682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:26:02.574280 13702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:26:02.576656 13708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:26:02.576722 13708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:26:02.574011 13539 master_runner.cc:186] Error getting master registration for 127.31.214.126:36387: OK, instance_id { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" instance_seqno: 1790576762145407 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:26:02.577142 13539 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 06:26:02.578321 13719 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:26:02.578395 13719 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:26:02.580157 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.571205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.31.214.124:41243)
0928 06:26:02.571262 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.571264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.571732 (+ 468us) server_negotiation.cc:407] Connection header received
0928 06:26:02.571827 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.571829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.571843 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.571874 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.572739 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.573524 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.574667 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.574970 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.575069 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.576725 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.576727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.576727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.576735 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.578611 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.579120 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.579121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.579122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.579188 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.579416 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.579418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.579418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.579576 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.579698 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.579955 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.580056 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:26:02.577886 13614 master_runner.cc:186] Error getting master registration for 127.31.214.126:36387: OK, instance_id { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" instance_seqno: 1790576762145407 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:26:02.580984 13614 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:26:02.582242 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.571124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.31.214.126:36387)
0928 06:26:02.571181 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.571183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.572902 (+ 1719us) server_negotiation.cc:407] Connection header received
0928 06:26:02.573085 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.573087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.573099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.573133 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.573927 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.574675 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.575703 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.576038 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.576127 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.578640 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.578642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.578642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.578651 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.580486 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.581282 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.581283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.581284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.581350 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.581589 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.581590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.581590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.581733 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.581807 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.582045 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.582142 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:26:02.587610 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.578714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.31.214.125:37421)
0928 06:26:02.578763 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.578766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.581005 (+ 2239us) server_negotiation.cc:407] Connection header received
0928 06:26:02.581110 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.581112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.581125 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.581158 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.581764 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.582518 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.583422 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.583637 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.583715 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.584927 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.584929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.584930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.584938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.586456 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.586807 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.586809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.586809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.586870 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.587075 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.587076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.587076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.587201 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.587328 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.587415 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.587492 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:26:02.593487 13532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } attempt: 3
Sep 28 06:26:02 dist-test-slave-q03f krb5kdc[13451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576762, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
I20260928 06:26:02.601326 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.594014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.31.214.124:41243)
0928 06:26:02.594054 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:02.594055 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:02.594063 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:26:02.594084 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:02.594085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:02.594100 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:02.594174 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:02.594525 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.595031 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.596175 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.596419 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.596504 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.598517 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:02.598519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:02.598520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:02.598528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:02.600016 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.600421 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.600422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.600423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.600510 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:02.600751 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:02.600752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:02.600752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:02.600881 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:02.600959 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:02.601147 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:26:02.601228 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:26:02.601326 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:02.593815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:41243 (local address 127.0.0.1:52142)
0928 06:26:02.593876 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:02.593885 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:02.593916 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:02.594181 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:02.594183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:02.594354 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:02.594397 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.594399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.595126 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.595128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.595778 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:02.595782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:02.596503 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:02.596504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:02.596567 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:02.597053 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:02.597058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:02.598422 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:02.600153 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.600155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.600155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.600344 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.600566 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:02.600567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:02.600568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:02.600646 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:02.600961 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:02.600962 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:02.601019 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:26:02.601101 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:26:02.601218 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:26:02.602979 13532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538: Bootstrap starting.
I20260928 06:26:02.603724 13532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:02.604117 13532 log.cc:824] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:02.606631 13532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538: No bootstrap required, opened a new log
I20260928 06:26:02.608273 13532 raft_consensus.cc:348] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } }
I20260928 06:26:02.608430 13532 raft_consensus.cc:374] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:02.608476 13532 raft_consensus.cc:729] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58d47cf40c8418d98fbc5b45ea5c538, State: Initialized, Role: FOLLOWER
I20260928 06:26:02.608645 13532 consensus_queue.cc:260] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } }
I20260928 06:26:02.608991 13724 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.609084 13532 sys_catalog.cc:564] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:26:02.609126 13724 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:26:02.614058 13464 master_runner.cc:263] No leader master found from master b58d47cf40c8418d98fbc5b45ea5c538
I20260928 06:26:02.614127 13464 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 06:26:02.614434 13735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:26:02.614559 13735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 06:26:02.643498 13485 tablet.cc:2369] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:26:02.703331 13702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:02.703487 13702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } }
I20260928 06:26:02.703770 13702 leader_election.cc:288] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b58d47cf40c8418d98fbc5b45ea5c538 (127.31.214.126:36387), 48b1923294684160ba9da7619c3d2af7 (127.31.214.125:37421)
I20260928 06:26:02.704335 13506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "318d98ffb35349b0979d62e24c6c5f04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" is_pre_election: true
I20260928 06:26:02.704515 13506 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 318d98ffb35349b0979d62e24c6c5f04 in term 0.
I20260928 06:26:02.704962 13633 leader_election.cc:302] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318d98ffb35349b0979d62e24c6c5f04, b58d47cf40c8418d98fbc5b45ea5c538; no voters:
I20260928 06:26:02.705174 13702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:26:02.705233 13702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:26:02.705258 13702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:02.706321 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "318d98ffb35349b0979d62e24c6c5f04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b1923294684160ba9da7619c3d2af7" is_pre_election: true
I20260928 06:26:02.706488 13581 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 318d98ffb35349b0979d62e24c6c5f04 in term 0.
I20260928 06:26:02.707258 13702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } }
I20260928 06:26:02.707418 13702 leader_election.cc:288] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [CANDIDATE]: Term 1 election: Requested vote from peers b58d47cf40c8418d98fbc5b45ea5c538 (127.31.214.126:36387), 48b1923294684160ba9da7619c3d2af7 (127.31.214.125:37421)
I20260928 06:26:02.707722 13506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "318d98ffb35349b0979d62e24c6c5f04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b58d47cf40c8418d98fbc5b45ea5c538"
I20260928 06:26:02.707823 13506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:02.708885 13506 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 318d98ffb35349b0979d62e24c6c5f04 in term 1.
I20260928 06:26:02.709151 13633 leader_election.cc:302] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318d98ffb35349b0979d62e24c6c5f04, b58d47cf40c8418d98fbc5b45ea5c538; no voters:
I20260928 06:26:02.709452 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "318d98ffb35349b0979d62e24c6c5f04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b1923294684160ba9da7619c3d2af7"
I20260928 06:26:02.709556 13581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:02.710687 13581 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 318d98ffb35349b0979d62e24c6c5f04 in term 1.
I20260928 06:26:02.711933 13702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:26:02.712095 13702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [term 1 LEADER]: Becoming Leader. State: Replica: 318d98ffb35349b0979d62e24c6c5f04, State: Running, Role: LEADER
I20260928 06:26:02.712186 13702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } }
I20260928 06:26:02.712574 13702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 318d98ffb35349b0979d62e24c6c5f04. Latest consensus state: current_term: 1 leader_uuid: "318d98ffb35349b0979d62e24c6c5f04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.712636 13702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:02.713109 13739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:26:02.713357 13739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:26:02.715619 13581 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [term 1 FOLLOWER]: Refusing update from remote peer 318d98ffb35349b0979d62e24c6c5f04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:02.716356 13737 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:02.718267 13695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 318d98ffb35349b0979d62e24c6c5f04. Latest consensus state: current_term: 1 leader_uuid: "318d98ffb35349b0979d62e24c6c5f04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.718391 13695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:26:02.718977 13506 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [term 1 FOLLOWER]: Refusing update from remote peer 318d98ffb35349b0979d62e24c6c5f04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:02.719275 13702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:02.721010 13724 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 318d98ffb35349b0979d62e24c6c5f04. Latest consensus state: current_term: 1 leader_uuid: "318d98ffb35349b0979d62e24c6c5f04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.721107 13724 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:26:02.722290 13737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "318d98ffb35349b0979d62e24c6c5f04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.722395 13737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:02.722892 13695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "318d98ffb35349b0979d62e24c6c5f04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.722983 13695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:26:02.723304 13740 mvcc.cc:205] Tried to move back new op lower bound from 7334202420080594944 to 7334202420069670912. Current Snapshot: MvccSnapshot[applied={T|T < 7334202420080594944}]
I20260928 06:26:02.723940 13741 mvcc.cc:205] Tried to move back new op lower bound from 7334202420080594944 to 7334202420069670912. Current Snapshot: MvccSnapshot[applied={T|T < 7334202420080594944}]
I20260928 06:26:02.725354 13739 catalog_manager.cc:1347] Generated new cluster ID: 2169a7a4fb494ca499159c7e622dbf6e
I20260928 06:26:02.725404 13739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:26:02.737888 13739 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:26:02.739014 13724 sys_catalog.cc:455] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "318d98ffb35349b0979d62e24c6c5f04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.739120 13724 sys_catalog.cc:458] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:26:02.739559 13743 mvcc.cc:205] Tried to move back new op lower bound from 7334202420080594944 to 7334202420069670912. Current Snapshot: MvccSnapshot[applied={T|T < 7334202420080594944}]
I20260928 06:26:02.738323 13737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "318d98ffb35349b0979d62e24c6c5f04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b58d47cf40c8418d98fbc5b45ea5c538" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 36387 } } peers { permanent_uuid: "48b1923294684160ba9da7619c3d2af7" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 37421 } } peers { permanent_uuid: "318d98ffb35349b0979d62e24c6c5f04" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 41243 } } }
I20260928 06:26:02.739877 13737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04 [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:02.740669 13739 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:26:02.750695 13739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 318d98ffb35349b0979d62e24c6c5f04: Generated new TSK 0
I20260928 06:26:02.750988 13739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:26:03.577207 13708 catalog_manager.cc:1259] Loaded cluster ID: 2169a7a4fb494ca499159c7e622dbf6e
I20260928 06:26:03.577339 13708 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7: loading cluster ID for follower catalog manager: success
I20260928 06:26:03.578267 13708 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7: acquiring CA information for follower catalog manager: success
I20260928 06:26:03.578661 13708 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 48b1923294684160ba9da7619c3d2af7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:26:03.615111 13735 catalog_manager.cc:1259] Loaded cluster ID: 2169a7a4fb494ca499159c7e622dbf6e
I20260928 06:26:03.615206 13735 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538: loading cluster ID for follower catalog manager: success
I20260928 06:26:03.616024 13735 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538: acquiring CA information for follower catalog manager: success
I20260928 06:26:03.616278 13735 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b58d47cf40c8418d98fbc5b45ea5c538: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:26:03.821221 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 13464
I20260928 06:26:03.827687 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 13539
I20260928 06:26:03.833670 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 13614
2026-09-28T06:26:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3059 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:26:03 dist-test-slave-q03f krb5kdc[13754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:26:03 dist-test-slave-q03f krb5kdc[13754](info): set up 2 sockets
Sep 28 06:26:03 dist-test-slave-q03f krb5kdc[13754](info): commencing operation
krb5kdc: starting...
W20260928 06:26:04.894218 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:26:04 dist-test-slave-q03f krb5kdc[13754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:26:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:26:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:26:05.027166 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35175
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:05.111632 13767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:05.111912 13767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:05.111938 13767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:05.113070 13767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:26:05.113111 13767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:05.113126 13767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:05.113139 13767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:26:05.113152 13767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:26:05.114643 13767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:35175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:05.114868 13767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:05.115052 13767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:05.120091 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:05.120095 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:05.120260 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:05.120618 13767 server_base.cc:1034] running on GCE node
I20260928 06:26:05.120918 13767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:05.121188 13767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:05.122334 13767 hybrid_clock.cc:648] HybridClock initialized: now 1790576765122301 us; error 48 us; skew 500 ppm
Sep 28 06:26:05 dist-test-slave-q03f krb5kdc[13754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576765, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:05.125142 13767 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:26:05.125667 13767 webserver.cc:460] Webserver started at http://127.31.214.126:45643/ using document root <none> and password file <none>
I20260928 06:26:05.126024 13767 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:05.126097 13767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:05.126266 13767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:05.127475 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb"
format_stamp: "Formatted at 2026-09-28 06:26:05 on dist-test-slave-q03f"
server_key: "691228bd344b17fc62b4365bf38d90fc"
server_key_iv: "37b98344cba9dcf9213cd2741eab7580"
server_key_version: "encryptionkey@0"
I20260928 06:26:05.127859 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb"
format_stamp: "Formatted at 2026-09-28 06:26:05 on dist-test-slave-q03f"
server_key: "691228bd344b17fc62b4365bf38d90fc"
server_key_iv: "37b98344cba9dcf9213cd2741eab7580"
server_key_version: "encryptionkey@0"
I20260928 06:26:05.129354 13767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:26:05.130255 13782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:05.130443 13767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:26:05.130555 13767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb"
format_stamp: "Formatted at 2026-09-28 06:26:05 on dist-test-slave-q03f"
server_key: "691228bd344b17fc62b4365bf38d90fc"
server_key_iv: "37b98344cba9dcf9213cd2741eab7580"
server_key_version: "encryptionkey@0"
I20260928 06:26:05.130635 13767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:05.147421 13767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:05.149597 13767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:05.149776 13767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:05.154225 13767 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:35175
I20260928 06:26:05.154285 13834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:35175 every 8 connection(s)
I20260928 06:26:05.154670 13767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:26:05.155732 13835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:05.158180 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb: Bootstrap starting.
I20260928 06:26:05.158820 13835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:05.159140 13835 log.cc:824] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:05.159978 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb: No bootstrap required, opened a new log
I20260928 06:26:05.161850 13835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35175 } }
I20260928 06:26:05.162127 13835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:05.162189 13835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9a3ecd1dcb416f8ba6241fb2407dbb, State: Initialized, Role: FOLLOWER
I20260928 06:26:05.162400 13835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35175 } }
I20260928 06:26:05.162513 13835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:26:05.162555 13835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:26:05.162563 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 13767
I20260928 06:26:05.162616 13835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:05.162665 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:26:05.162905 32601 external_mini_cluster.cc:1428] Setting key 433802971e613dd6489e1c71d9a7bad6
I20260928 06:26:05.163486 13835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35175 } }
I20260928 06:26:05.163620 13835 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c9a3ecd1dcb416f8ba6241fb2407dbb; no voters:
I20260928 06:26:05.163861 13835 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:26:05.163905 13840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:26:05.164103 13840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [term 1 LEADER]: Becoming Leader. State: Replica: 3c9a3ecd1dcb416f8ba6241fb2407dbb, State: Running, Role: LEADER
I20260928 06:26:05.164260 13840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35175 } }
I20260928 06:26:05.164326 13835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:26:05.164662 13840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9a3ecd1dcb416f8ba6241fb2407dbb. Latest consensus state: current_term: 1 leader_uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35175 } } }
I20260928 06:26:05.164822 13840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:05.164848 13842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9a3ecd1dcb416f8ba6241fb2407dbb" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35175 } } }
I20260928 06:26:05.165019 13842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:05.165120 13848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:26:05.165735 13848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:26:05.167483 13848 catalog_manager.cc:1347] Generated new cluster ID: 5722acd1d6384db5b6464a71fab8564f
I20260928 06:26:05.167544 13848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:26:05 dist-test-slave-q03f krb5kdc[13754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:05.175237 13839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:05.163489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.31.214.126:35175)
0928 06:26:05.163757 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:05.163770 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:05.163800 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:26:05.163949 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:05.163957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:05.164025 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:05.165221 (+ 1196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:05.165367 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:05.166851 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:05.167924 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:05.168543 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:05.168653 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:05.171201 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:05.171206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:05.171206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:05.171221 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:05.173310 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:05.173746 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:05.173750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:05.173750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:05.173883 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:05.174141 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:05.174142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:05.174142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:05.174462 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:05.174620 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:05.174755 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:26:05.174891 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":176,"threads_started":1}
I20260928 06:26:05.176780 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:26:05.187633 13848 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:26:05.188426 13848 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:26:05.192180 13848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c9a3ecd1dcb416f8ba6241fb2407dbb: Generated new TSK 0
I20260928 06:26:05.192397 13848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:26:05.231248 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 13767
2026-09-28T06:26:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1395 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:26:05 dist-test-slave-q03f krb5kdc[13862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:26:05 dist-test-slave-q03f krb5kdc[13862](info): set up 2 sockets
Sep 28 06:26:05 dist-test-slave-q03f krb5kdc[13862](info): commencing operation
krb5kdc: starting...
W20260928 06:26:06.294591 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:26:06 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:26:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:26:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:26:06.425774 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35673
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:06.509145 13875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:06.509367 13875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:06.509392 13875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:06.510596 13875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:26:06.510640 13875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:06.510654 13875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:06.510667 13875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:26:06.510679 13875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:26:06.512130 13875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:06.512337 13875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:06.512516 13875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:06.517904 13883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:06.517923 13880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:06.517923 13881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:06.518018 13875 server_base.cc:1034] running on GCE node
I20260928 06:26:06.518653 13875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:06.518961 13875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:06.520147 13875 hybrid_clock.cc:648] HybridClock initialized: now 1790576766520153 us; error 57 us; skew 500 ppm
Sep 28 06:26:06 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:06.533324 13875 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:26:06.533995 13875 webserver.cc:460] Webserver started at http://127.31.214.126:38575/ using document root <none> and password file <none>
I20260928 06:26:06.534269 13875 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:06.534343 13875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:06.534519 13875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:06.535679 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "5597bb59e5f44a08b91f8b4de37a53ef"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "32cb83ad1e9e7ba4fb101e011786d646"
server_key_iv: "12d55c442986754aa5d05e32eb9c3c50"
server_key_version: "encryptionkey@0"
I20260928 06:26:06.536083 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "5597bb59e5f44a08b91f8b4de37a53ef"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "32cb83ad1e9e7ba4fb101e011786d646"
server_key_iv: "12d55c442986754aa5d05e32eb9c3c50"
server_key_version: "encryptionkey@0"
I20260928 06:26:06.537564 13875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:26:06.538480 13890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:06.538697 13875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:26:06.538815 13875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "5597bb59e5f44a08b91f8b4de37a53ef"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "32cb83ad1e9e7ba4fb101e011786d646"
server_key_iv: "12d55c442986754aa5d05e32eb9c3c50"
server_key_version: "encryptionkey@0"
I20260928 06:26:06.538895 13875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:06.558815 13875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:06.561149 13875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:06.561327 13875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:06.565842 13875 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:35673
I20260928 06:26:06.565878 13942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:35673 every 8 connection(s)
I20260928 06:26:06.566278 13875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:26:06.567337 13943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:06.569777 13943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Bootstrap starting.
I20260928 06:26:06.570437 13943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:06.570737 13943 log.cc:824] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:06.571578 13943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: No bootstrap required, opened a new log
I20260928 06:26:06.571658 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 13875
I20260928 06:26:06.571784 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:26:06.572005 32601 external_mini_cluster.cc:1428] Setting key 18e1a98734b4518ed13a342b3dacfc6c
I20260928 06:26:06.574131 13943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:06.574321 13943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:06.574361 13943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5597bb59e5f44a08b91f8b4de37a53ef, State: Initialized, Role: FOLLOWER
I20260928 06:26:06.574496 13943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:06.574576 13943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:26:06.574599 13943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:26:06.574654 13943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:06.575757 13943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:06.575871 13943 leader_election.cc:302] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5597bb59e5f44a08b91f8b4de37a53ef; no voters:
I20260928 06:26:06.576020 13943 leader_election.cc:288] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:26:06.576090 13948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:26:06.576262 13948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 1 LEADER]: Becoming Leader. State: Replica: 5597bb59e5f44a08b91f8b4de37a53ef, State: Running, Role: LEADER
I20260928 06:26:06.576290 13943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:26:06.576357 13948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:06.576735 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } } }
I20260928 06:26:06.576830 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:06.577086 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5597bb59e5f44a08b91f8b4de37a53ef. Latest consensus state: current_term: 1 leader_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } } }
I20260928 06:26:06.577543 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:06.577477 13956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:26:06.578562 13956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:26:06.580399 13956 catalog_manager.cc:1347] Generated new cluster ID: 1416fa9e225a4bfcae25a40e1454b78a
I20260928 06:26:06.580473 13956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:26:06 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:06.585659 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:06.572650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.31.214.126:35673)
0928 06:26:06.572861 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:06.572871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:06.572905 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:26:06.574079 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:06.574090 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:06.574161 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:06.574327 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:06.575206 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.576672 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.577843 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.578530 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.578679 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:06.581592 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:06.581596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:06.581596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:06.581609 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:06.583813 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:06.584266 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:06.584269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:06.584269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:06.584403 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:06.584641 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:06.584641 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:06.584642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:06.584945 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:06.585105 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:06.585297 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:26:06.585443 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260928 06:26:06.597558 13956 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:26:06.598232 13956 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:26:06.601528 13956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Generated new TSK 0
I20260928 06:26:06.601761 13956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:26:06.630349 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35673
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:06.723529 13969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:06.723784 13969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:06.723814 13969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:06.725136 13969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:06.725185 13969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:06.725216 13969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:26:06.727078 13969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:06.727320 13969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:06.727548 13969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:06.733410 13974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:06.733701 13969 server_base.cc:1034] running on GCE node
W20260928 06:26:06.733497 13975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:06.733405 13977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:06.734181 13969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:06.734481 13969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:06.735667 13969 hybrid_clock.cc:648] HybridClock initialized: now 1790576766735659 us; error 46 us; skew 500 ppm
Sep 28 06:26:06 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:06.738674 13969 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:26:06.739180 13969 webserver.cc:460] Webserver started at http://127.31.214.65:44451/ using document root <none> and password file <none>
I20260928 06:26:06.739431 13969 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:06.739501 13969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:06.739655 13969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:06.740724 13969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "14eca66fbead497ba67a704c3697bee2"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cf0261fe0d3282f9f9c53e5c03480971"
server_key_iv: "990ae51e9144b993b84fc1b89d76da3d"
server_key_version: "encryptionkey@0"
I20260928 06:26:06.741115 13969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "14eca66fbead497ba67a704c3697bee2"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cf0261fe0d3282f9f9c53e5c03480971"
server_key_iv: "990ae51e9144b993b84fc1b89d76da3d"
server_key_version: "encryptionkey@0"
I20260928 06:26:06.742627 13969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:26:06.743542 13984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:06.743775 13969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:06.743886 13969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "14eca66fbead497ba67a704c3697bee2"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cf0261fe0d3282f9f9c53e5c03480971"
server_key_iv: "990ae51e9144b993b84fc1b89d76da3d"
server_key_version: "encryptionkey@0"
I20260928 06:26:06.743978 13969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:06.753067 13969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:06.755095 13969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:06.755270 13969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:06.755501 13969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:06.755824 13969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:06.755878 13969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:06.755913 13969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:06.755941 13969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:06.762221 13969 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:37417
I20260928 06:26:06.762301 14097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:37417 every 8 connection(s)
I20260928 06:26:06.762662 13969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:26:06.766176 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 13969
I20260928 06:26:06.766338 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:26:06.766556 32601 external_mini_cluster.cc:1428] Setting key e5284bd42718a8d3d3ef14762962235b
Sep 28 06:26:06 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:06.774271 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:06.763517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:44955 (local address 127.31.214.126:35673)
0928 06:26:06.763565 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:06.763567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:06.763771 (+ 204us) server_negotiation.cc:407] Connection header received
0928 06:26:06.764278 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:06.764279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:06.764293 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:06.764366 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:06.765308 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.765876 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.767438 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.767828 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.767909 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:06.770996 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:06.770997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:06.770998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:06.771008 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:06.772628 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:06.773117 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:06.773118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:06.773119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:06.773184 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:06.773485 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:06.773492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:06.773492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:06.773626 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:06.773706 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:06.774068 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 06:26:06.774169 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:26:06.774685 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:06.763482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:44955)
0928 06:26:06.763634 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:06.763682 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:06.764186 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:06.764421 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:06.764424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:06.764991 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:06.765214 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:06.765219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.766190 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.766191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:06.767336 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:06.767340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.767927 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.767929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:06.768050 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:06.768696 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:06.768723 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:06.770872 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:06.772743 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:06.772753 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:06.772755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:06.773033 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:06.773262 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:06.773263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:06.773264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:06.773406 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:06.773700 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:06.773708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:06.773867 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 06:26:06.774042 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 06:26:06.774231 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260928 06:26:06.775393 14098 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35673
I20260928 06:26:06.775522 14098 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:06.775749 14098 heartbeater.cc:511] Master 127.31.214.126:35673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
I20260928 06:26:06.776412 13907 ts_manager.cc:194] Registered new tserver with Master: 14eca66fbead497ba67a704c3697bee2 (127.31.214.65:37417)
I20260928 06:26:06.777175 13907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:44955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:26:06.801575 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35673
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:06.889737 14103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:06.889976 14103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:06.890004 14103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:06.891237 14103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:06.891283 14103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:06.891311 14103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:26:06.892840 14103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:06.893234 14103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:06.893507 14103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:06.899278 14111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:06.899326 14103 server_base.cc:1034] running on GCE node
W20260928 06:26:06.899281 14108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:06.899304 14109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:06.899698 14103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:06.899981 14103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:06.901149 14103 hybrid_clock.cc:648] HybridClock initialized: now 1790576766901141 us; error 54 us; skew 500 ppm
Sep 28 06:26:06 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:06.903955 14103 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:26:06.904434 14103 webserver.cc:460] Webserver started at http://127.31.214.66:39765/ using document root <none> and password file <none>
I20260928 06:26:06.904685 14103 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:06.904754 14103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:06.904908 14103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:06.906091 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "66e6758f854d4098915bd11da97738f1"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cdd40a976efb54ef18c392d358633cc3"
server_key_iv: "df2ec757c92ec52b4c9e5abdcb37d1f9"
server_key_version: "encryptionkey@0"
I20260928 06:26:06.906474 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "66e6758f854d4098915bd11da97738f1"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cdd40a976efb54ef18c392d358633cc3"
server_key_iv: "df2ec757c92ec52b4c9e5abdcb37d1f9"
server_key_version: "encryptionkey@0"
I20260928 06:26:06.907871 14103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:26:06.908674 14118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:06.908862 14103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:06.908972 14103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "66e6758f854d4098915bd11da97738f1"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cdd40a976efb54ef18c392d358633cc3"
server_key_iv: "df2ec757c92ec52b4c9e5abdcb37d1f9"
server_key_version: "encryptionkey@0"
I20260928 06:26:06.909049 14103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:06.937112 14103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:06.939762 14103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:06.939939 14103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:06.940237 14103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:06.940579 14103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:06.940642 14103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:06.940707 14103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:06.940738 14103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:06.947088 14103 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:40967
I20260928 06:26:06.947153 14231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:40967 every 8 connection(s)
I20260928 06:26:06.947475 14103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:26:06 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576766, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:06.957556 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 14103
I20260928 06:26:06.957683 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:26:06.957904 32601 external_mini_cluster.cc:1428] Setting key e7fe20bd44d17ec532e9b8f9724916e9
I20260928 06:26:06.958105 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:06.948303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:56997 (local address 127.31.214.126:35673)
0928 06:26:06.948382 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:06.948384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:06.948629 (+ 245us) server_negotiation.cc:407] Connection header received
0928 06:26:06.949125 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:06.949126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:06.949137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:06.949264 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:06.950491 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.951142 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.952139 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.952355 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.952427 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:06.954825 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:06.954826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:06.954827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:06.954833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:06.956483 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:06.956920 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:06.956922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:06.956922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:06.956985 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:06.957286 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:06.957287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:06.957288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:06.957426 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:06.957524 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:06.957879 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 06:26:06.958008 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:26:06.958351 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:06.948290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:56997)
0928 06:26:06.948469 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:06.948514 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:06.949030 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:06.949350 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:06.949353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:06.950043 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:06.950366 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:06.950371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.951246 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.951247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:06.952041 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:06.952043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:06.952484 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:06.952485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:06.952614 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:06.953218 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:06.953226 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:06.954734 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:06.956594 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:06.956601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:06.956603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:06.956825 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:06.957053 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:06.957054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:06.957054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:06.957211 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:06.957541 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:06.957551 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:06.957720 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 06:26:06.957856 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 06:26:06.958056 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260928 06:26:06.958877 14232 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35673
I20260928 06:26:06.958995 14232 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:06.959187 14232 heartbeater.cc:511] Master 127.31.214.126:35673 requested a full tablet report, sending...
I20260928 06:26:06.959599 13907 ts_manager.cc:194] Registered new tserver with Master: 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967)
I20260928 06:26:06.959988 13907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:56997
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:26:06.986147 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35673
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:07.071266 14237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:07.071482 14237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:07.071507 14237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:07.072690 14237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:07.072734 14237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:07.072762 14237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:26:07.074271 14237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:07.074532 14237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:07.074772 14237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:07.080394 14242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:07.080394 14243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:07.080457 14245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:07.080499 14237 server_base.cc:1034] running on GCE node
I20260928 06:26:07.080727 14237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:07.080940 14237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:07.082072 14237 hybrid_clock.cc:648] HybridClock initialized: now 1790576767082052 us; error 35 us; skew 500 ppm
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:07.084940 14237 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:26:07.085577 14237 webserver.cc:460] Webserver started at http://127.31.214.67:41145/ using document root <none> and password file <none>
I20260928 06:26:07.085867 14237 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:07.085932 14237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:07.086102 14237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:07.087167 14237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "05ed5564fc464b39a61ac6a1a7523de8"
format_stamp: "Formatted at 2026-09-28 06:26:07 on dist-test-slave-q03f"
server_key: "09b6d2e6f76d26034fc9bedb89640d5a"
server_key_iv: "a6e8a269eb48c52d32b722ff063f6b84"
server_key_version: "encryptionkey@0"
I20260928 06:26:07.087466 14237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "05ed5564fc464b39a61ac6a1a7523de8"
format_stamp: "Formatted at 2026-09-28 06:26:07 on dist-test-slave-q03f"
server_key: "09b6d2e6f76d26034fc9bedb89640d5a"
server_key_iv: "a6e8a269eb48c52d32b722ff063f6b84"
server_key_version: "encryptionkey@0"
I20260928 06:26:07.088779 14237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:26:07.089586 14252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:07.089794 14237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:07.089905 14237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "05ed5564fc464b39a61ac6a1a7523de8"
format_stamp: "Formatted at 2026-09-28 06:26:07 on dist-test-slave-q03f"
server_key: "09b6d2e6f76d26034fc9bedb89640d5a"
server_key_iv: "a6e8a269eb48c52d32b722ff063f6b84"
server_key_version: "encryptionkey@0"
I20260928 06:26:07.090021 14237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:07.130034 14237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:07.132515 14237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:07.132711 14237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:07.133015 14237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:07.133411 14237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:07.133452 14237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:07.133495 14237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:07.133522 14237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:07.139839 14237 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:38017
I20260928 06:26:07.139904 14365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:38017 every 8 connection(s)
I20260928 06:26:07.140264 14237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:26:07.142300 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 14237
I20260928 06:26:07.142397 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:26:07.142582 32601 external_mini_cluster.cc:1428] Setting key 239cf8ccdd470c2965e394f1a34e2770
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:07.151058 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.141099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:52877 (local address 127.31.214.126:35673)
0928 06:26:07.141163 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.141165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.141428 (+ 263us) server_negotiation.cc:407] Connection header received
0928 06:26:07.142174 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.142175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.142189 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.142254 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:07.143666 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.144206 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.145236 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.145436 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.145539 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.147978 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:07.147979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:07.147980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:07.147994 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:07.149543 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.149997 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.149998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.149999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.150089 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.150358 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.150359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.150359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.150481 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:07.150553 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.150857 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.150953 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:26:07.151237 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.141087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:52877)
0928 06:26:07.141291 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.141344 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.142054 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.142383 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.142390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.143239 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:07.143564 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.143571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.144304 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.144306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.145141 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.145143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.145554 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.145555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.145697 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.146337 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:07.146347 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:07.147879 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:07.149649 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:07.149656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:07.149659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:07.149893 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:07.150162 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:07.150163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:07.150164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:07.150283 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:07.150563 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:07.150571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:07.150714 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.150819 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.150986 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260928 06:26:07.151798 14366 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35673
I20260928 06:26:07.151906 14366 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:07.152057 14366 heartbeater.cc:511] Master 127.31.214.126:35673 requested a full tablet report, sending...
I20260928 06:26:07.152633 13907 ts_manager.cc:194] Registered new tserver with Master: 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017)
I20260928 06:26:07.153496 13907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:52877
I20260928 06:26:07.154651 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:26:07.162566 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.156094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.31.214.126:35673)
0928 06:26:07.156267 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.156269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.156358 (+ 89us) server_negotiation.cc:407] Connection header received
0928 06:26:07.156441 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.156442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.156452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.156524 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:07.157093 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.157587 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.158397 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.158591 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.158677 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.159928 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:07.159930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:07.159930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:07.159938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:07.161338 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.161700 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.161701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.161702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.161763 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.161973 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.161974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.161974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.162144 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:07.162254 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.162364 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.162448 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260928 06:26:07.164284 13907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:26:07.166003 13907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:26:07.172904 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.168575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:40967 (local address 127.0.0.1:35848)
0928 06:26:07.168999 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.169009 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.169083 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.169428 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.169429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.169431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:07.169490 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.169492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.170854 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.170855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.171904 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.171906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.172558 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.172559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.172645 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.172660 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.172734 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.172784 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260928 06:26:07.172914 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.168690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.31.214.66:40967)
0928 06:26:07.168950 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.168955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.169175 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:26:07.169233 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.169235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.169250 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.169327 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:07.169573 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.170742 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.171969 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.172461 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.172557 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.172621 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.172781 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.172826 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":123,"threads_started":1}
I20260928 06:26:07.173085 14379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.168679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.31.214.67:38017)
0928 06:26:07.168977 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.168982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.169180 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:26:07.169227 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.169229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.169247 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.169326 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:07.169604 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.170459 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.171929 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.172461 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.172551 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.172592 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.172953 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.173007 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":90,"threads_started":1}
I20260928 06:26:07.172914 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.168569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:38017 (local address 127.0.0.1:57016)
0928 06:26:07.168931 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.168978 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.169084 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.169400 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.169404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.169406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:07.169495 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.169497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.170561 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.170562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.171832 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.171835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.172644 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.172645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.172693 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.172697 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.172774 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.172819 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":67,"threads_started":1}
I20260928 06:26:07.173776 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.168794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:37417 (local address 127.0.0.1:47882)
0928 06:26:07.169130 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.169141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.169189 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.169700 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.169700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.169702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:07.169734 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.169735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.171009 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.171010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.171867 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.171870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.173514 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.173516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.173580 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.173585 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.173637 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.173687 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260928 06:26:07.173842 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.168918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.31.214.65:37417)
0928 06:26:07.169460 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.169465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.169476 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:26:07.169516 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.169519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.169536 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.169610 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:07.169821 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.170905 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.172659 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.173423 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.173499 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.173551 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.173713 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.173760 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":108,"threads_started":1}
I20260928 06:26:07.173915 14166 tablet_service.cc:1467] Processing CreateTablet for tablet 99bb46454486410cbe2a7835314c11af (DEFAULT_TABLE table=test-table [id=a5bfc8a3eb6d4d58960401a82decf505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:07.174376 14166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bb46454486410cbe2a7835314c11af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:07.174374 14300 tablet_service.cc:1467] Processing CreateTablet for tablet 99bb46454486410cbe2a7835314c11af (DEFAULT_TABLE table=test-table [id=a5bfc8a3eb6d4d58960401a82decf505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:07.174486 14032 tablet_service.cc:1467] Processing CreateTablet for tablet 99bb46454486410cbe2a7835314c11af (DEFAULT_TABLE table=test-table [id=a5bfc8a3eb6d4d58960401a82decf505]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:07.174744 14300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bb46454486410cbe2a7835314c11af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:07.174811 14032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bb46454486410cbe2a7835314c11af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:07.177713 14383 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: Bootstrap starting.
I20260928 06:26:07.178134 14385 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Bootstrap starting.
I20260928 06:26:07.178377 14384 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Bootstrap starting.
I20260928 06:26:07.178774 14383 tablet_bootstrap.cc:654] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:07.178773 14385 tablet_bootstrap.cc:654] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:07.179005 14384 tablet_bootstrap.cc:654] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:07.179400 14385 log.cc:824] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:07.179400 14383 log.cc:824] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:07.179579 14384 log.cc:824] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:07.180418 14383 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: No bootstrap required, opened a new log
I20260928 06:26:07.180410 14384 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: No bootstrap required, opened a new log
I20260928 06:26:07.180488 14383 ts_tablet_manager.cc:1397] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:26:07.180488 14384 ts_tablet_manager.cc:1397] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:26:07.180595 14385 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: No bootstrap required, opened a new log
I20260928 06:26:07.180667 14385 ts_tablet_manager.cc:1397] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:26:07.182116 14385 raft_consensus.cc:348] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.182291 14385 raft_consensus.cc:374] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:07.182337 14385 raft_consensus.cc:729] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14eca66fbead497ba67a704c3697bee2, State: Initialized, Role: FOLLOWER
I20260928 06:26:07.182483 14385 consensus_queue.cc:260] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.182548 14384 raft_consensus.cc:348] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.182587 14383 raft_consensus.cc:348] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.182718 14383 raft_consensus.cc:374] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:07.182718 14384 raft_consensus.cc:374] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:07.182760 14384 raft_consensus.cc:729] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e6758f854d4098915bd11da97738f1, State: Initialized, Role: FOLLOWER
I20260928 06:26:07.182760 14383 raft_consensus.cc:729] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ed5564fc464b39a61ac6a1a7523de8, State: Initialized, Role: FOLLOWER
I20260928 06:26:07.182879 14384 consensus_queue.cc:260] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.182972 14385 ts_tablet_manager.cc:1428] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:26:07.183003 14098 heartbeater.cc:503] Master 127.31.214.126:35673 was elected leader, sending a full tablet report...
I20260928 06:26:07.183197 14384 ts_tablet_manager.cc:1428] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:26:07.182894 14383 consensus_queue.cc:260] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.183327 14232 heartbeater.cc:503] Master 127.31.214.126:35673 was elected leader, sending a full tablet report...
I20260928 06:26:07.183501 14383 ts_tablet_manager.cc:1428] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:26:07.183539 14366 heartbeater.cc:503] Master 127.31.214.126:35673 was elected leader, sending a full tablet report...
W20260928 06:26:07.198233 14233 tablet.cc:2369] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:26:07.263458 14099 tablet.cc:2369] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:26:07.315826 14390 raft_consensus.cc:482] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:07.315948 14390 raft_consensus.cc:504] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.316514 14390 leader_election.cc:288] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017), 14eca66fbead497ba67a704c3697bee2 (127.31.214.65:37417)
I20260928 06:26:07.319530 14379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.316876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:52005 (local address 127.31.214.67:38017)
0928 06:26:07.316936 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.316938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.316946 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:26:07.316980 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.316981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.316989 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.317052 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:07.317228 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.317737 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.318777 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.319170 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.319260 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.319270 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.319423 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.319460 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:26:07.319530 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.316698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:38017 (local address 127.31.214.66:52005)
0928 06:26:07.316792 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.316806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.316859 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.317077 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.317079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.317081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:07.317159 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.317160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.317849 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.317851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.318678 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.318680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.319257 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.319258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.319345 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.319348 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.319415 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.319456 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 06:26:07.319628 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.316831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:53795 (local address 127.31.214.65:37417)
0928 06:26:07.316882 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.316884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.317006 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:26:07.317091 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.317092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.317100 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.317161 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:07.317325 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.317806 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.318890 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.319252 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.319339 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.319347 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.319499 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.319533 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:26:07.319664 14392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.316723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:37417 (local address 127.31.214.66:53795)
0928 06:26:07.316911 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.316922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.317011 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.317164 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.317165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.317167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:07.317253 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.317254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.317929 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.317930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.318787 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.318790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.319341 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.319343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.319415 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.319421 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.319492 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.319531 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":113,"threads_started":1}
I20260928 06:26:07.319994 14052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bb46454486410cbe2a7835314c11af" candidate_uuid: "66e6758f854d4098915bd11da97738f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14eca66fbead497ba67a704c3697bee2" is_pre_election: true
I20260928 06:26:07.320082 14320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bb46454486410cbe2a7835314c11af" candidate_uuid: "66e6758f854d4098915bd11da97738f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ed5564fc464b39a61ac6a1a7523de8" is_pre_election: true
I20260928 06:26:07.320182 14052 raft_consensus.cc:2454] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e6758f854d4098915bd11da97738f1 in term 0.
I20260928 06:26:07.320258 14320 raft_consensus.cc:2454] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e6758f854d4098915bd11da97738f1 in term 0.
I20260928 06:26:07.320446 14119 leader_election.cc:302] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14eca66fbead497ba67a704c3697bee2, 66e6758f854d4098915bd11da97738f1; no voters:
I20260928 06:26:07.320583 14390 raft_consensus.cc:2781] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:26:07.320650 14390 raft_consensus.cc:482] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:26:07.320677 14390 raft_consensus.cc:3037] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:07.321398 14390 raft_consensus.cc:504] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.321568 14390 leader_election.cc:288] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [CANDIDATE]: Term 1 election: Requested vote from peers 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017), 14eca66fbead497ba67a704c3697bee2 (127.31.214.65:37417)
I20260928 06:26:07.321732 14320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bb46454486410cbe2a7835314c11af" candidate_uuid: "66e6758f854d4098915bd11da97738f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ed5564fc464b39a61ac6a1a7523de8"
I20260928 06:26:07.321743 14052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bb46454486410cbe2a7835314c11af" candidate_uuid: "66e6758f854d4098915bd11da97738f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14eca66fbead497ba67a704c3697bee2"
I20260928 06:26:07.321813 14052 raft_consensus.cc:3037] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:07.321813 14320 raft_consensus.cc:3037] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:07.322861 14052 raft_consensus.cc:2454] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e6758f854d4098915bd11da97738f1 in term 1.
I20260928 06:26:07.322861 14320 raft_consensus.cc:2454] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e6758f854d4098915bd11da97738f1 in term 1.
I20260928 06:26:07.323037 14119 leader_election.cc:302] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14eca66fbead497ba67a704c3697bee2, 66e6758f854d4098915bd11da97738f1; no voters:
I20260928 06:26:07.323136 14390 raft_consensus.cc:2781] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:26:07.323323 14390 raft_consensus.cc:686] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 1 LEADER]: Becoming Leader. State: Replica: 66e6758f854d4098915bd11da97738f1, State: Running, Role: LEADER
I20260928 06:26:07.323422 14390 consensus_queue.cc:237] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.324347 13907 catalog_manager.cc:5495] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e6758f854d4098915bd11da97738f1 (127.31.214.66). New cstate: current_term: 1 leader_uuid: "66e6758f854d4098915bd11da97738f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } health_report { overall_health: UNKNOWN } } }
I20260928 06:26:07.355188 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.352680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.31.214.66:40967)
0928 06:26:07.352780 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.352781 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.352790 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:26:07.352829 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.352830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.352849 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.352917 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:26:07.353041 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.353527 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.354403 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.354590 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.354644 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.354815 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:26:07.354920 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:26:07.354983 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.355017 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.355036 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 06:26:07.357282 14320 raft_consensus.cc:1261] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [term 1 FOLLOWER]: Refusing update from remote peer 66e6758f854d4098915bd11da97738f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:07.357282 14052 raft_consensus.cc:1261] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Refusing update from remote peer 66e6758f854d4098915bd11da97738f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:07.357714 14390 consensus_queue.cc:1035] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:07.357933 14393 consensus_queue.cc:1035] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:07.363977 13906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:26:07.364094 13906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:26:07.364909 13906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Sending DeleteTablet for 3 replicas of tablet 99bb46454486410cbe2a7835314c11af
I20260928 06:26:07.365285 14300 tablet_service.cc:1514] Processing DeleteTablet for tablet 99bb46454486410cbe2a7835314c11af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:07 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:57016
I20260928 06:26:07.365299 14032 tablet_service.cc:1514] Processing DeleteTablet for tablet 99bb46454486410cbe2a7835314c11af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:07 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:47882
I20260928 06:26:07.365367 14166 tablet_service.cc:1514] Processing DeleteTablet for tablet 99bb46454486410cbe2a7835314c11af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:07 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:35848
I20260928 06:26:07.365393 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 13969
I20260928 06:26:07.366375 14405 tablet_replica.cc:330] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: stopping tablet replica
I20260928 06:26:07.366503 14405 raft_consensus.cc:2229] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:26:07.366621 14405 raft_consensus.cc:2258] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:07.367436 14405 ts_tablet_manager.cc:1905] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:07.369537 14405 ts_tablet_manager.cc:1918] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:07.369620 14405 log.cc:1196] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/99bb46454486410cbe2a7835314c11af
I20260928 06:26:07.369894 14405 ts_tablet_manager.cc:1939] T 99bb46454486410cbe2a7835314c11af P 05ed5564fc464b39a61ac6a1a7523de8: Deleting consensus metadata
I20260928 06:26:07.370429 13894 catalog_manager.cc:4841] TS 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017): tablet 99bb46454486410cbe2a7835314c11af (table test-table [id=a5bfc8a3eb6d4d58960401a82decf505]) successfully deleted
W20260928 06:26:07.372558 13891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:37417 (error 108)
I20260928 06:26:07.372627 14407 tablet_replica.cc:330] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: stopping tablet replica
I20260928 06:26:07.372748 14407 raft_consensus.cc:2229] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:26:07.372747 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 14103
I20260928 06:26:07.372879 14407 raft_consensus.cc:2258] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:07.374732 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.374582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:37417 (local address 127.0.0.1:47888)
0928 06:26:07.374645 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.374691 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:37417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:26:07.374871 13891 catalog_manager.cc:4584] TS 14eca66fbead497ba67a704c3697bee2 (127.31.214.65:37417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99bb46454486410cbe2a7835314c11af: Network error: Client connection negotiation failed: client connection to 127.31.214.65:37417: connect: Connection refused (error 111)
I20260928 06:26:07.378716 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.378547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:40967 (local address 127.0.0.1:35862)
0928 06:26:07.378642 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.378678 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.66:40967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:26:07.378918 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 14237
I20260928 06:26:07.384682 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 13875
I20260928 06:26:07.390419 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35673
--webserver_interface=127.31.214.126
--webserver_port=38575
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:07.475901 14409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:07.476147 14409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:07.476186 14409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:07.477396 14409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:26:07.477476 14409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:07.477500 14409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:07.477521 14409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:26:07.477563 14409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:26:07.479206 14409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=38575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:07.479523 14409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:07.479773 14409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:07.490262 14415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:07.490319 14416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:07.490295 14418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:07.490651 14409 server_base.cc:1034] running on GCE node
I20260928 06:26:07.490988 14409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:07.491343 14409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:07.492522 14409 hybrid_clock.cc:648] HybridClock initialized: now 1790576767492504 us; error 42 us; skew 500 ppm
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:07.495386 14409 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:26:07.495930 14409 webserver.cc:460] Webserver started at http://127.31.214.126:38575/ using document root <none> and password file <none>
I20260928 06:26:07.496184 14409 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:07.496290 14409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:07.497797 14409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:26:07.498611 14425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:07.498806 14409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:26:07.498919 14409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "5597bb59e5f44a08b91f8b4de37a53ef"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "32cb83ad1e9e7ba4fb101e011786d646"
server_key_iv: "12d55c442986754aa5d05e32eb9c3c50"
server_key_version: "encryptionkey@0"
I20260928 06:26:07.499398 14409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:07.529742 14409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:07.532189 14409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:07.532397 14409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:07.536867 14409 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:35673
I20260928 06:26:07.536906 14477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:35673 every 8 connection(s)
I20260928 06:26:07.537343 14409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:26:07.539165 14478 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:26:07.539795 14478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Bootstrap starting.
I20260928 06:26:07.546069 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 14409
I20260928 06:26:07.546447 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:37417
--local_ip_for_outbound_sockets=127.31.214.65
--tserver_master_addrs=127.31.214.126:35673
--webserver_port=44451
--webserver_interface=127.31.214.65
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:07.549369 14478 log.cc:824] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:07.551707 14478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:26:07.552029 14478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Bootstrap complete.
I20260928 06:26:07.554544 14478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:07.554807 14478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5597bb59e5f44a08b91f8b4de37a53ef, State: Initialized, Role: FOLLOWER
I20260928 06:26:07.554982 14478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:07.555038 14478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:26:07.555101 14478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:26:07.555153 14478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:26:07.556702 14478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:07.556838 14478 leader_election.cc:302] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5597bb59e5f44a08b91f8b4de37a53ef; no voters:
I20260928 06:26:07.557039 14478 leader_election.cc:288] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:26:07.557075 14482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:26:07.557197 14482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 2 LEADER]: Becoming Leader. State: Replica: 5597bb59e5f44a08b91f8b4de37a53ef, State: Running, Role: LEADER
I20260928 06:26:07.557293 14482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:07.557328 14478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:26:07.557560 14482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5597bb59e5f44a08b91f8b4de37a53ef. Latest consensus state: current_term: 2 leader_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } } }
I20260928 06:26:07.557638 14482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:07.557564 14483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } } }
I20260928 06:26:07.557816 14483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:07.558341 14486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:26:07.559175 14486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:26:07.559283 14486 catalog_manager.cc:1259] Loaded cluster ID: 1416fa9e225a4bfcae25a40e1454b78a
I20260928 06:26:07.559319 14486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:26:07.559841 14486 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 06:26:07.565006 14486 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 06:26:07.567875 14486 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:26:07.568105 14486 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Loaded TSK: 0
I20260928 06:26:07.568337 14486 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:26:07.636744 14480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:07.636955 14480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:07.636979 14480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:07.638280 14480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:07.638329 14480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:07.638361 14480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:26:07.639916 14480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:37417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=44451
--tserver_master_addrs=127.31.214.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:07.640169 14480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:07.640436 14480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:07.645733 14505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:07.645761 14508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:07.645745 14506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:07.646167 14480 server_base.cc:1034] running on GCE node
I20260928 06:26:07.646401 14480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:07.646695 14480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:07.647883 14480 hybrid_clock.cc:648] HybridClock initialized: now 1790576767647868 us; error 27 us; skew 500 ppm
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:07.650970 14480 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:26:07.651510 14480 webserver.cc:460] Webserver started at http://127.31.214.65:44451/ using document root <none> and password file <none>
I20260928 06:26:07.651779 14480 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:07.651858 14480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:07.653647 14480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:26:07.654528 14515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:07.654744 14480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:26:07.654847 14480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "14eca66fbead497ba67a704c3697bee2"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cf0261fe0d3282f9f9c53e5c03480971"
server_key_iv: "990ae51e9144b993b84fc1b89d76da3d"
server_key_version: "encryptionkey@0"
I20260928 06:26:07.655243 14480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:07.677489 14480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:07.680011 14480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:07.680186 14480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:07.680446 14480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:07.680928 14522 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:26:07.682976 14480 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:26:07.683040 14480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:26:07.683087 14480 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:26:07.683769 14480 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:26:07.683825 14480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:26:07.683918 14522 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Bootstrap starting.
I20260928 06:26:07.691668 14480 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:37417
I20260928 06:26:07.692093 14480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:26:07.692772 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 14480
I20260928 06:26:07.693089 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:40967
--local_ip_for_outbound_sockets=127.31.214.66
--tserver_master_addrs=127.31.214.126:35673
--webserver_port=39765
--webserver_interface=127.31.214.66
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:07.696033 14522 log.cc:824] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:07.698838 14522 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:26:07.699247 14522 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Bootstrap complete.
I20260928 06:26:07.700573 14522 ts_tablet_manager.cc:1397] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260928 06:26:07.701056 14629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:37417 every 8 connection(s)
I20260928 06:26:07.702128 14522 raft_consensus.cc:348] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.702464 14522 raft_consensus.cc:729] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14eca66fbead497ba67a704c3697bee2, State: Initialized, Role: FOLLOWER
I20260928 06:26:07.702736 14522 consensus_queue.cc:260] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.703126 14522 ts_tablet_manager.cc:1428] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:07.710201 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.693038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:33803)
0928 06:26:07.696368 (+ 3330us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.696419 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.697207 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.697940 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.697962 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.698398 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:07.698754 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.698763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.701276 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.701278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.702933 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.702938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.703982 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.703983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.704097 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.704750 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:07.704760 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:07.706294 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:07.708506 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:07.708514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:07.708516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:07.708752 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:07.709038 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:07.709038 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:07.709039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:07.709152 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:07.709627 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:07.709632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:07.709758 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.709838 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.710034 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3279,"thread_start_us":104,"threads_started":1}
I20260928 06:26:07.710307 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.693130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:33803 (local address 127.31.214.126:35673)
0928 06:26:07.697446 (+ 4316us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.697458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.697501 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:26:07.697643 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.697650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.697720 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.698956 (+ 1236us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:07.699061 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.700637 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.703086 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.703850 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.703989 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.706429 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:07.706435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:07.706435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:07.706448 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:07.708414 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.708847 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.708850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.708851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.708970 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.709224 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.709225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.709225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.709549 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:07.709643 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.709933 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.710106 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4251,"thread_start_us":96,"threads_started":1}
I20260928 06:26:07.711162 14630 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35673
I20260928 06:26:07.711287 14630 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:07.711529 14630 heartbeater.cc:511] Master 127.31.214.126:35673 requested a full tablet report, sending...
I20260928 06:26:07.712421 14442 ts_manager.cc:194] Registered new tserver with Master: 14eca66fbead497ba67a704c3697bee2 (127.31.214.65:37417)
I20260928 06:26:07.713442 14442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:33803
I20260928 06:26:07.714524 14630 heartbeater.cc:503] Master 127.31.214.126:35673 was elected leader, sending a full tablet report...
I20260928 06:26:07.717370 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.713388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.31.214.65:37417)
0928 06:26:07.714071 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.714075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.714083 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:26:07.714160 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.714162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.714379 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.714473 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:07.714704 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.715419 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.716455 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.716936 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.717027 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.717087 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.717245 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.717286 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":93,"threads_started":1}
I20260928 06:26:07.717388 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.713102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:37417 (local address 127.0.0.1:47904)
0928 06:26:07.713301 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.713338 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.713411 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.714495 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.714498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.714500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:07.714567 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.714568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.715509 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.715510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.716360 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.716362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.717027 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.717028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.717115 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.717128 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.717226 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.717288 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":111,"threads_started":1}
I20260928 06:26:07.717864 14546 tablet_service.cc:1514] Processing DeleteTablet for tablet 99bb46454486410cbe2a7835314c11af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:07 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:47904
I20260928 06:26:07.718472 14639 tablet_replica.cc:330] stopping tablet replica
I20260928 06:26:07.718595 14639 raft_consensus.cc:2229] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:26:07.718667 14639 raft_consensus.cc:2258] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:07.719033 14639 ts_tablet_manager.cc:1905] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:07.721567 14639 ts_tablet_manager.cc:1918] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:07.721648 14639 log.cc:1196] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/99bb46454486410cbe2a7835314c11af
I20260928 06:26:07.721876 14639 ts_tablet_manager.cc:1939] T 99bb46454486410cbe2a7835314c11af P 14eca66fbead497ba67a704c3697bee2: Deleting consensus metadata
I20260928 06:26:07.722535 14426 catalog_manager.cc:4841] TS 14eca66fbead497ba67a704c3697bee2 (127.31.214.65:37417): tablet 99bb46454486410cbe2a7835314c11af (table test-table [id=a5bfc8a3eb6d4d58960401a82decf505]) successfully deleted
W20260928 06:26:07.787133 14634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:07.787356 14634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:07.787380 14634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:07.788527 14634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:07.788566 14634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:07.788594 14634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:26:07.790406 14634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:40967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=39765
--tserver_master_addrs=127.31.214.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:07.790617 14634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:07.790809 14634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:07.796097 14644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:07.796147 14634 server_base.cc:1034] running on GCE node
W20260928 06:26:07.796085 14645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:07.796171 14647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:07.796558 14634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:07.796814 14634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:07.798019 14634 hybrid_clock.cc:648] HybridClock initialized: now 1790576767798010 us; error 31 us; skew 500 ppm
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:07.800645 14634 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:26:07.801200 14634 webserver.cc:460] Webserver started at http://127.31.214.66:39765/ using document root <none> and password file <none>
I20260928 06:26:07.801453 14634 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:07.801556 14634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:07.803274 14634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:07.804106 14654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:07.804307 14634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:07.804411 14634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "66e6758f854d4098915bd11da97738f1"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cdd40a976efb54ef18c392d358633cc3"
server_key_iv: "df2ec757c92ec52b4c9e5abdcb37d1f9"
server_key_version: "encryptionkey@0"
I20260928 06:26:07.804849 14634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:07.820032 14634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:07.822408 14634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:07.822590 14634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:07.822892 14634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:07.823381 14661 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:26:07.824929 14634 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:26:07.824982 14634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:26:07.825011 14634 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:26:07.825760 14634 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:26:07.825798 14634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:26:07.825903 14661 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Bootstrap starting.
I20260928 06:26:07.834450 14634 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:40967
I20260928 06:26:07.834702 14768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:40967 every 8 connection(s)
I20260928 06:26:07.834753 14661 log.cc:824] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:07.834959 14634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:26:07.837527 14661 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:26:07.837828 14661 tablet_bootstrap.cc:492] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Bootstrap complete.
I20260928 06:26:07.838863 14661 ts_tablet_manager.cc:1397] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260928 06:26:07.839906 14661 raft_consensus.cc:348] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.840317 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 14634
I20260928 06:26:07.840577 14661 raft_consensus.cc:729] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e6758f854d4098915bd11da97738f1, State: Initialized, Role: FOLLOWER
I20260928 06:26:07.840622 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:38017
--local_ip_for_outbound_sockets=127.31.214.67
--tserver_master_addrs=127.31.214.126:35673
--webserver_port=41145
--webserver_interface=127.31.214.67
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:07.840782 14661 consensus_queue.cc:260] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:07.841099 14661 ts_tablet_manager.cc:1428] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:07.846421 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.835864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:35413 (local address 127.31.214.126:35673)
0928 06:26:07.835913 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.835915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.836218 (+ 303us) server_negotiation.cc:407] Connection header received
0928 06:26:07.836732 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.836734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.836747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.836820 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:07.837521 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.838276 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.839487 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.839743 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.839835 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.842847 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:07.842849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:07.842850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:07.842857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:07.844565 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.845173 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.845174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.845175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.845232 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.845592 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.845593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.845593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.845740 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:07.845799 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.846263 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.846348 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:26:07.846736 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.835803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:35413)
0928 06:26:07.836001 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.836041 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.836616 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.836894 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.836898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.837181 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:07.837397 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.837403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.838382 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.838384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.839380 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.839383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.839848 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.839849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.839982 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.840639 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:07.840648 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:07.842725 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:07.844679 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:07.844689 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:07.844693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:07.845090 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:07.845306 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:07.845307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:07.845308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:07.845500 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:07.845818 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:07.845825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:07.846087 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.846223 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.846438 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260928 06:26:07.847360 14770 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35673
I20260928 06:26:07.847482 14770 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:07.847707 14770 heartbeater.cc:511] Master 127.31.214.126:35673 requested a full tablet report, sending...
I20260928 06:26:07.848395 14442 ts_manager.cc:194] Registered new tserver with Master: 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967)
I20260928 06:26:07.849040 14442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:35413
I20260928 06:26:07.850015 14770 heartbeater.cc:503] Master 127.31.214.126:35673 was elected leader, sending a full tablet report...
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.66@KRBTEST.COM
I20260928 06:26:07.857641 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.848751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:40967 (local address 127.0.0.1:35866)
0928 06:26:07.849183 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.849195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.849284 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.849497 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.849498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.849799 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:07.849836 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.849839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.850959 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.850961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.851477 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.851479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.852061 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.852062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:07.852131 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.852711 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:07.852716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:07.854281 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:07.856489 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:07.856497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:07.856499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:07.856788 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:07.857006 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:07.857006 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:07.857007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:07.857067 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:07.857381 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:07.857388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:07.857433 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:26:07.857487 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:26:07.857570 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414}
I20260928 06:26:07.857787 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.848841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.31.214.66:40967)
0928 06:26:07.849048 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.849054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.849280 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:26:07.849361 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.849363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.849376 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.849457 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:07.849927 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.850876 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.851566 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.851976 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:07.852063 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:07.854372 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:07.854380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:07.854381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:07.854389 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:07.856412 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.856862 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.856865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.856866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.856933 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:07.857128 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:07.857129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:07.857129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:07.857316 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:07.857400 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:07.857544 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:26:07.857644 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
I20260928 06:26:07.858202 14703 tablet_service.cc:1514] Processing DeleteTablet for tablet 99bb46454486410cbe2a7835314c11af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:07 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:35866
I20260928 06:26:07.858939 14776 tablet_replica.cc:330] stopping tablet replica
I20260928 06:26:07.859078 14776 raft_consensus.cc:2229] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:26:07.859167 14776 raft_consensus.cc:2258] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:07.859549 14776 ts_tablet_manager.cc:1905] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:07.861770 14776 ts_tablet_manager.cc:1918] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:07.861872 14776 log.cc:1196] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/99bb46454486410cbe2a7835314c11af
I20260928 06:26:07.862159 14776 ts_tablet_manager.cc:1939] T 99bb46454486410cbe2a7835314c11af P 66e6758f854d4098915bd11da97738f1: Deleting consensus metadata
I20260928 06:26:07.863610 14426 catalog_manager.cc:4841] TS 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967): tablet 99bb46454486410cbe2a7835314c11af (table test-table [id=a5bfc8a3eb6d4d58960401a82decf505]) successfully deleted
W20260928 06:26:07.936043 14774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:07.936276 14774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:07.936302 14774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:07.937577 14774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:07.937634 14774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:07.937664 14774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:26:07.939322 14774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:38017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=41145
--tserver_master_addrs=127.31.214.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:07.939579 14774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:07.939857 14774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:26:07.945549 14774 server_base.cc:1034] running on GCE node
W20260928 06:26:07.945617 14784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:07.945509 14782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:07.945518 14781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:07.946007 14774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:07.946301 14774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:07.947485 14774 hybrid_clock.cc:648] HybridClock initialized: now 1790576767947478 us; error 35 us; skew 500 ppm
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:07.950584 14774 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:26:07.951149 14774 webserver.cc:460] Webserver started at http://127.31.214.67:41145/ using document root <none> and password file <none>
I20260928 06:26:07.951366 14774 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:07.951426 14774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:07.953014 14774 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:07.953717 14791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:07.953923 14774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:07.954061 14774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "05ed5564fc464b39a61ac6a1a7523de8"
format_stamp: "Formatted at 2026-09-28 06:26:07 on dist-test-slave-q03f"
server_key: "09b6d2e6f76d26034fc9bedb89640d5a"
server_key_iv: "a6e8a269eb48c52d32b722ff063f6b84"
server_key_version: "encryptionkey@0"
I20260928 06:26:07.954437 14774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:07.984197 14774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:07.987308 14774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:07.987504 14774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:07.987800 14774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:07.988161 14774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:07.988222 14774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:07.988263 14774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:07.988317 14774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:07.994918 14774 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:38017
I20260928 06:26:07.995090 14904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:38017 every 8 connection(s)
I20260928 06:26:07.995416 14774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:26:07.997213 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 14774
Sep 28 06:26:07 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:08.008471 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.997742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.31.214.126:35673)
0928 06:26:07.997903 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.997905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.998195 (+ 290us) server_negotiation.cc:407] Connection header received
0928 06:26:07.998316 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.998317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.998329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.998408 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:07.999262 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.999968 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.001692 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.002074 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.002145 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.004165 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:08.004167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:08.004167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:08.004175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:08.006234 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:08.006702 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:08.006703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:08.006704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:08.006806 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:08.007546 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:08.007547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:08.007548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:08.007747 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:08.007913 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:08.008033 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:26:08.008169 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260928 06:26:08.009279 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.996475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:51393 (local address 127.31.214.126:35673)
0928 06:26:07.996546 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:07.996548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:07.996585 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:26:07.997993 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:07.997994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:07.998006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:07.998061 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:07.999141 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:07.999958 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.001010 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.001367 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.001434 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.004857 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:08.004859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:08.004859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:08.004869 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:08.007131 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:08.007731 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:08.007732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:08.007732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:08.007812 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:08.008212 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:08.008213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:08.008213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:08.008369 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:08.008418 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:08.008877 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 06:26:08.009177 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:26:08.009385 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:07.996218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:51393)
0928 06:26:07.996429 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:07.996484 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:07.997868 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:07.998137 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:07.998141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:07.998616 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:07.998973 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:07.998980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.000083 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.000084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:08.000918 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:08.000920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.001460 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.001461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:08.001638 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.002322 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:08.002331 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:08.004680 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:08.007233 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:08.007244 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:08.007247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:08.007657 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:08.007898 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:08.007899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:08.007899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:08.008120 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:08.008466 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:08.008473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:08.008690 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 06:26:08.008849 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 06:26:08.009101 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260928 06:26:08.009889 14905 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35673
I20260928 06:26:08.010044 14905 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:08.010259 14905 heartbeater.cc:511] Master 127.31.214.126:35673 requested a full tablet report, sending...
I20260928 06:26:08.010699 14442 ts_manager.cc:194] Registered new tserver with Master: 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017)
I20260928 06:26:08.011137 14442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:51393
I20260928 06:26:08.021009 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:26:08.024223 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.021489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.31.214.126:35673)
0928 06:26:08.021553 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:08.021555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:08.021640 (+ 85us) server_negotiation.cc:407] Connection header received
0928 06:26:08.021698 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:08.021698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:08.021757 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:08.021826 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:26:08.022026 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.022512 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.023521 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.023736 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.023793 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.023933 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:26:08.024034 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:26:08.024096 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:08.024141 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:26:08.024158 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:26:08.024566 14442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:26:08.026345 14442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:26:08.028577 14546 tablet_service.cc:1467] Processing CreateTablet for tablet f9a614f1d2064227800af942deb2773b (DEFAULT_TABLE table=test-table [id=ea7017d048ad4cefbdff183b5c6c8393]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:08.028684 14703 tablet_service.cc:1467] Processing CreateTablet for tablet f9a614f1d2064227800af942deb2773b (DEFAULT_TABLE table=test-table [id=ea7017d048ad4cefbdff183b5c6c8393]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:08.028803 14546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a614f1d2064227800af942deb2773b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:08.028908 14703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a614f1d2064227800af942deb2773b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:08.030392 14522 tablet_bootstrap.cc:492] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Bootstrap starting.
I20260928 06:26:08.030736 14522 tablet_bootstrap.cc:654] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:08.030781 14661 tablet_bootstrap.cc:492] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: Bootstrap starting.
I20260928 06:26:08.031211 14661 tablet_bootstrap.cc:654] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:08.031435 14522 tablet_bootstrap.cc:492] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: No bootstrap required, opened a new log
I20260928 06:26:08.031479 14522 ts_tablet_manager.cc:1397] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:08.031599 14522 raft_consensus.cc:348] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.031663 14522 raft_consensus.cc:374] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:08.031693 14522 raft_consensus.cc:729] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14eca66fbead497ba67a704c3697bee2, State: Initialized, Role: FOLLOWER
I20260928 06:26:08.031757 14522 consensus_queue.cc:260] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.031873 14522 ts_tablet_manager.cc:1428] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:08.031934 14661 tablet_bootstrap.cc:492] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: No bootstrap required, opened a new log
I20260928 06:26:08.031977 14661 ts_tablet_manager.cc:1397] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:26:08.032364 14661 raft_consensus.cc:348] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.032429 14661 raft_consensus.cc:374] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:08.032507 14661 raft_consensus.cc:729] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e6758f854d4098915bd11da97738f1, State: Initialized, Role: FOLLOWER
I20260928 06:26:08.032570 14661 consensus_queue.cc:260] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.032680 14661 ts_tablet_manager.cc:1428] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:26:08.033170 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.028325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:38017 (local address 127.0.0.1:57028)
0928 06:26:08.028389 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.028401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:08.028439 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:08.028940 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:08.028941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:08.028944 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:08.028996 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:08.028998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.030499 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.030501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:08.031738 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:08.031741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.032904 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.032905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:08.032984 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.032988 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:26:08.033041 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:26:08.033089 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:26:08.033454 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.028430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.31.214.67:38017)
0928 06:26:08.028710 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:08.028716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:08.028728 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:26:08.028773 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:08.028775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:08.028804 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:08.028937 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:08.029072 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.030358 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.031852 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.032717 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.032795 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.032865 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:08.033312 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 06:26:08.033366 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":136,"threads_started":1}
I20260928 06:26:08.034368 14839 tablet_service.cc:1467] Processing CreateTablet for tablet f9a614f1d2064227800af942deb2773b (DEFAULT_TABLE table=test-table [id=ea7017d048ad4cefbdff183b5c6c8393]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:08.034808 14839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a614f1d2064227800af942deb2773b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:08.038126 14914 tablet_bootstrap.cc:492] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: Bootstrap starting.
I20260928 06:26:08.038923 14914 tablet_bootstrap.cc:654] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:08.039484 14914 log.cc:824] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:08.040370 14914 tablet_bootstrap.cc:492] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: No bootstrap required, opened a new log
I20260928 06:26:08.040462 14914 ts_tablet_manager.cc:1397] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:26:08.041847 14914 raft_consensus.cc:348] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.042074 14914 raft_consensus.cc:374] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:08.042136 14914 raft_consensus.cc:729] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ed5564fc464b39a61ac6a1a7523de8, State: Initialized, Role: FOLLOWER
I20260928 06:26:08.042305 14914 consensus_queue.cc:260] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.042647 14914 ts_tablet_manager.cc:1428] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:26:08.042717 14905 heartbeater.cc:503] Master 127.31.214.126:35673 was elected leader, sending a full tablet report...
W20260928 06:26:08.085654 14771 tablet.cc:2369] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:26:08.174532 14636 raft_consensus.cc:482] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:08.174678 14636 raft_consensus.cc:504] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.175015 14636 leader_election.cc:288] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017), 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967)
I20260928 06:26:08.178583 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.175402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:57235 (local address 127.31.214.66:40967)
0928 06:26:08.175453 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:08.175455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:08.175472 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:26:08.175536 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:08.175537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:08.175566 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:08.175644 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:08.175939 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.176675 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.177708 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.178210 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.178301 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.178394 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:08.178456 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:26:08.178496 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:26:08.178627 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.175283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:40967 (local address 127.31.214.65:57235)
0928 06:26:08.175421 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.175430 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:08.175476 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:08.175669 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:08.175670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:08.175671 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:08.175821 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:08.175824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.176776 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.176778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:08.177607 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:08.177610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.178321 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.178322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:08.178390 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.178393 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:26:08.178440 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:26:08.178487 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":71,"thread_start_us":74,"threads_started":1}
I20260928 06:26:08.178997 14723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a614f1d2064227800af942deb2773b" candidate_uuid: "14eca66fbead497ba67a704c3697bee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e6758f854d4098915bd11da97738f1" is_pre_election: true
I20260928 06:26:08.179200 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.175280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:38017 (local address 127.31.214.65:45307)
0928 06:26:08.175374 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.175391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:08.175462 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:08.175668 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:08.175668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:08.175670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:08.176155 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:08.176159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.177059 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.177061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:08.178020 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:08.178022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.178903 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.178905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:08.178988 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.178993 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:26:08.179062 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:26:08.179129 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":3}
I20260928 06:26:08.179211 14723 raft_consensus.cc:2454] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14eca66fbead497ba67a704c3697bee2 in term 0.
I20260928 06:26:08.179253 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.175451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:45307 (local address 127.31.214.67:38017)
0928 06:26:08.175494 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:08.175497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:08.175506 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:26:08.175536 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:08.175537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:08.175578 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:08.175643 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:08.176285 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.176811 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.178152 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.178613 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.178722 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.178734 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:08.179108 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 06:26:08.179185 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:26:08.179467 14516 leader_election.cc:302] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14eca66fbead497ba67a704c3697bee2, 66e6758f854d4098915bd11da97738f1; no voters:
I20260928 06:26:08.179623 14636 raft_consensus.cc:2781] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:26:08.179592 14859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a614f1d2064227800af942deb2773b" candidate_uuid: "14eca66fbead497ba67a704c3697bee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ed5564fc464b39a61ac6a1a7523de8" is_pre_election: true
I20260928 06:26:08.179687 14636 raft_consensus.cc:482] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:26:08.179714 14636 raft_consensus.cc:3037] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:08.179765 14859 raft_consensus.cc:2454] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14eca66fbead497ba67a704c3697bee2 in term 0.
I20260928 06:26:08.180532 14636 raft_consensus.cc:504] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.180702 14636 leader_election.cc:288] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 1 election: Requested vote from peers 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017), 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967)
I20260928 06:26:08.180938 14723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a614f1d2064227800af942deb2773b" candidate_uuid: "14eca66fbead497ba67a704c3697bee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e6758f854d4098915bd11da97738f1"
I20260928 06:26:08.180953 14859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a614f1d2064227800af942deb2773b" candidate_uuid: "14eca66fbead497ba67a704c3697bee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ed5564fc464b39a61ac6a1a7523de8"
I20260928 06:26:08.181025 14723 raft_consensus.cc:3037] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:08.181034 14859 raft_consensus.cc:3037] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:08.181804 14723 raft_consensus.cc:2454] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14eca66fbead497ba67a704c3697bee2 in term 1.
I20260928 06:26:08.181895 14859 raft_consensus.cc:2454] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14eca66fbead497ba67a704c3697bee2 in term 1.
I20260928 06:26:08.182006 14516 leader_election.cc:302] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14eca66fbead497ba67a704c3697bee2, 66e6758f854d4098915bd11da97738f1; no voters:
I20260928 06:26:08.182168 14636 raft_consensus.cc:2781] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:26:08.182322 14636 raft_consensus.cc:686] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 LEADER]: Becoming Leader. State: Replica: 14eca66fbead497ba67a704c3697bee2, State: Running, Role: LEADER
I20260928 06:26:08.182387 14636 consensus_queue.cc:237] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.183324 14442 catalog_manager.cc:5495] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14eca66fbead497ba67a704c3697bee2 (127.31.214.65). New cstate: current_term: 1 leader_uuid: "14eca66fbead497ba67a704c3697bee2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } health_report { overall_health: HEALTHY } } }
W20260928 06:26:08.204821 14631 tablet.cc:2369] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:26:08.217779 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.215114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.31.214.65:37417)
0928 06:26:08.215192 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:08.215194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:08.215202 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:26:08.215241 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:08.215241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:08.215261 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:08.215326 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:26:08.215431 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.215938 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.216784 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:08.216986 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:08.217045 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:08.217343 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:26:08.217457 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:26:08.217532 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:08.217690 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:26:08.217714 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:26:08.219549 14859 raft_consensus.cc:1261] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [term 1 FOLLOWER]: Refusing update from remote peer 14eca66fbead497ba67a704c3697bee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:08.219578 14723 raft_consensus.cc:1261] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Refusing update from remote peer 14eca66fbead497ba67a704c3697bee2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:08.219990 14918 consensus_queue.cc:1035] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:08.220208 14636 consensus_queue.cc:1035] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:08.222994 14923 mvcc.cc:205] Tried to move back new op lower bound from 7334202442624532480 to 7334202442475933696. Current Snapshot: MvccSnapshot[applied={T|T < 7334202442624532480}]
I20260928 06:26:08.224781 14442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:26:08.224877 14442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:26:08.225708 14442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Sending DeleteTablet for 3 replicas of tablet f9a614f1d2064227800af942deb2773b
I20260928 06:26:08.226068 14839 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a614f1d2064227800af942deb2773b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:08 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:57028
I20260928 06:26:08.226105 14703 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a614f1d2064227800af942deb2773b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:08 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:35866
I20260928 06:26:08.226213 14776 tablet_replica.cc:330] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: stopping tablet replica
I20260928 06:26:08.226279 14776 raft_consensus.cc:2229] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:26:08.226303 14930 tablet_replica.cc:330] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: stopping tablet replica
I20260928 06:26:08.226342 14776 raft_consensus.cc:2258] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:08.226384 14930 raft_consensus.cc:2229] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:26:08.226428 14546 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a614f1d2064227800af942deb2773b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:08 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:47904
I20260928 06:26:08.226509 14930 raft_consensus.cc:2258] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:08.226744 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 14480
I20260928 06:26:08.226826 14776 ts_tablet_manager.cc:1905] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:08.227000 14930 ts_tablet_manager.cc:1905] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:08.228711 14776 ts_tablet_manager.cc:1918] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:08.228770 14776 log.cc:1196] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/f9a614f1d2064227800af942deb2773b
I20260928 06:26:08.228900 14930 ts_tablet_manager.cc:1918] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:08.228955 14930 log.cc:1196] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/f9a614f1d2064227800af942deb2773b
I20260928 06:26:08.229028 14776 ts_tablet_manager.cc:1939] T f9a614f1d2064227800af942deb2773b P 66e6758f854d4098915bd11da97738f1: Deleting consensus metadata
I20260928 06:26:08.229202 14930 ts_tablet_manager.cc:1939] T f9a614f1d2064227800af942deb2773b P 05ed5564fc464b39a61ac6a1a7523de8: Deleting consensus metadata
I20260928 06:26:08.229429 14426 catalog_manager.cc:4841] TS 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967): tablet f9a614f1d2064227800af942deb2773b (table test-table [id=ea7017d048ad4cefbdff183b5c6c8393]) successfully deleted
I20260928 06:26:08.229831 14429 catalog_manager.cc:4841] TS 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017): tablet f9a614f1d2064227800af942deb2773b (table test-table [id=ea7017d048ad4cefbdff183b5c6c8393]) successfully deleted
W20260928 06:26:08.233685 14426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:37417 (error 108)
I20260928 06:26:08.234222 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 14634
I20260928 06:26:08.234429 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.234163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:37417 (local address 127.0.0.1:47920)
0928 06:26:08.234308 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.234384 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:37417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 06:26:08.234948 14426 catalog_manager.cc:4584] TS 14eca66fbead497ba67a704c3697bee2 (127.31.214.65:37417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9a614f1d2064227800af942deb2773b: Network error: Client connection negotiation failed: client connection to 127.31.214.65:37417: connect: Connection refused (error 111)
I20260928 06:26:08.239811 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 14774
I20260928 06:26:08.245038 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 14409
I20260928 06:26:08.250720 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35673
--webserver_interface=127.31.214.126
--webserver_port=38575
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:08.334818 14933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:08.335088 14933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:08.335136 14933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:08.336468 14933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:26:08.336545 14933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:08.336570 14933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:08.336602 14933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:26:08.336650 14933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:26:08.338289 14933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=38575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:08.338589 14933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:08.338836 14933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:08.344048 14939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:08.344223 14938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:08.344030 14941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:08.344390 14933 server_base.cc:1034] running on GCE node
I20260928 06:26:08.344627 14933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:08.344915 14933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:08.346019 14933 hybrid_clock.cc:648] HybridClock initialized: now 1790576768346005 us; error 35 us; skew 500 ppm
Sep 28 06:26:08 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576768, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:08.348834 14933 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:26:08.349438 14933 webserver.cc:460] Webserver started at http://127.31.214.126:38575/ using document root <none> and password file <none>
I20260928 06:26:08.349694 14933 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:08.349766 14933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:08.351382 14933 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:08.352166 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:08.352478 14933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:08.352591 14933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "5597bb59e5f44a08b91f8b4de37a53ef"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "32cb83ad1e9e7ba4fb101e011786d646"
server_key_iv: "12d55c442986754aa5d05e32eb9c3c50"
server_key_version: "encryptionkey@0"
I20260928 06:26:08.352973 14933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:08.365458 14933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:08.367899 14933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:08.368083 14933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:08.372740 14933 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:35673
I20260928 06:26:08.372784 15000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:35673 every 8 connection(s)
I20260928 06:26:08.373186 14933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:26:08.375002 15001 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:26:08.375774 15001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Bootstrap starting.
I20260928 06:26:08.375983 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 14933
I20260928 06:26:08.376312 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:37417
--local_ip_for_outbound_sockets=127.31.214.65
--tserver_master_addrs=127.31.214.126:35673
--webserver_port=44451
--webserver_interface=127.31.214.65
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:08.385610 15001 log.cc:824] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:08.388196 15001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:26:08.388540 15001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef: Bootstrap complete.
I20260928 06:26:08.390997 15001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:08.391233 15001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5597bb59e5f44a08b91f8b4de37a53ef, State: Initialized, Role: FOLLOWER
I20260928 06:26:08.391404 15001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:08.391476 15001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:26:08.391546 15001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:26:08.391592 15001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:26:08.392678 15001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:08.392794 15001 leader_election.cc:302] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5597bb59e5f44a08b91f8b4de37a53ef; no voters:
I20260928 06:26:08.393010 15001 leader_election.cc:288] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:26:08.393050 15005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:26:08.393178 15005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [term 3 LEADER]: Becoming Leader. State: Replica: 5597bb59e5f44a08b91f8b4de37a53ef, State: Running, Role: LEADER
I20260928 06:26:08.393283 15005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } }
I20260928 06:26:08.393314 15001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:26:08.393721 15007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5597bb59e5f44a08b91f8b4de37a53ef. Latest consensus state: current_term: 3 leader_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } } }
I20260928 06:26:08.393801 15007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:08.394068 15009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:26:08.394565 15006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5597bb59e5f44a08b91f8b4de37a53ef" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35673 } } }
I20260928 06:26:08.394659 15006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5597bb59e5f44a08b91f8b4de37a53ef [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:08.394975 15009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:26:08.395089 15009 catalog_manager.cc:1259] Loaded cluster ID: 1416fa9e225a4bfcae25a40e1454b78a
I20260928 06:26:08.395403 15009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 06:26:08.395692 15009 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6afb3d09ad google::LogMessage::Fail() at ??:0
@ 0x7f6afb3d388a google::LogMessage::SendToLog() at ??:0
@ 0x7f6afb3d04de google::LogMessage::Flush() at ??:0
@ 0x7f6afb3d1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6b032f10f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6b032fa8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6afd29e0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6afd293954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6afe0f76db start_thread at ??:0
@ 0x7f6af9ece61f clone at ??:0
W20260928 06:26:08.489410 15003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:08.489641 15003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:08.489669 15003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:08.490943 15003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:08.490989 15003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:08.491019 15003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:26:08.492533 15003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:37417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=44451
--tserver_master_addrs=127.31.214.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:08.492794 15003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:08.493044 15003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:08.498458 15037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:08.498468 15038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:08.498440 15040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:08.499019 15003 server_base.cc:1034] running on GCE node
I20260928 06:26:08.499264 15003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:08.499569 15003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:08.500762 15003 hybrid_clock.cc:648] HybridClock initialized: now 1790576768500715 us; error 64 us; skew 500 ppm
Sep 28 06:26:08 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576768, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:08.504294 15003 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:26:08.504884 15003 webserver.cc:460] Webserver started at http://127.31.214.65:44451/ using document root <none> and password file <none>
I20260928 06:26:08.505167 15003 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:08.505245 15003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:08.506919 15003 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:08.507747 15047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:08.507975 15003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:08.508165 15003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "14eca66fbead497ba67a704c3697bee2"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cf0261fe0d3282f9f9c53e5c03480971"
server_key_iv: "990ae51e9144b993b84fc1b89d76da3d"
server_key_version: "encryptionkey@0"
I20260928 06:26:08.508764 15003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:08.531296 15003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:08.534307 15003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:08.534505 15003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:08.534783 15003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:08.535389 15054 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:26:08.537474 15003 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:26:08.537601 15003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:26:08.537729 15003 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:26:08.538460 15003 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:26:08.538517 15003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260928 06:26:08.538594 15054 tablet_bootstrap.cc:492] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Bootstrap starting.
I20260928 06:26:08.546208 15003 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:37417
I20260928 06:26:08.546377 15161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:37417 every 8 connection(s)
I20260928 06:26:08.546705 15003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:26:08.550359 15054 log.cc:824] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:08.552415 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 15003
I20260928 06:26:08.552709 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:40967
--local_ip_for_outbound_sockets=127.31.214.66
--tserver_master_addrs=127.31.214.126:35673
--webserver_port=39765
--webserver_interface=127.31.214.66
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:08.553542 15054 tablet_bootstrap.cc:492] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:26:08.553892 15054 tablet_bootstrap.cc:492] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Bootstrap complete.
I20260928 06:26:08.555325 15054 ts_tablet_manager.cc:1397] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.014s
I20260928 06:26:08.556875 15054 raft_consensus.cc:348] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.557237 15054 raft_consensus.cc:729] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14eca66fbead497ba67a704c3697bee2, State: Initialized, Role: FOLLOWER
I20260928 06:26:08.557427 15054 consensus_queue.cc:260] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:08.557758 15054 ts_tablet_manager.cc:1428] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260928 06:26:08.606621 15164 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:26:08.547421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:59505)
0928 06:26:08.547600 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.547655 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:08.548170 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:08.606312 (+ 58142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
W20260928 06:26:08.606957 15162 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:35673 (0 consecutive failures): Network error: Failed to ping master at 127.31.214.126:35673: Client connection negotiation failed: client connection to 127.31.214.126:35673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:26:08.607409 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.607258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:54339)
0928 06:26:08.607325 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.607382 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:26:08.607874 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.607759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:59063)
0928 06:26:08.607812 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.607851 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:26:08.607999 15162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:26:08.642123 15166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:08.642439 15166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:08.642481 15166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:08.643733 15166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:08.643806 15166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:08.643853 15166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:26:08.645363 15166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:40967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=39765
--tserver_master_addrs=127.31.214.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:08.645646 15166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:08.645884 15166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:08.651108 15173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:08.651121 15172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:08.651121 15175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:08.651386 15166 server_base.cc:1034] running on GCE node
I20260928 06:26:08.651625 15166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:08.651857 15166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:08.653043 15166 hybrid_clock.cc:648] HybridClock initialized: now 1790576768653034 us; error 36 us; skew 500 ppm
Sep 28 06:26:08 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576768, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:08.655977 15166 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:26:08.656507 15166 webserver.cc:460] Webserver started at http://127.31.214.66:39765/ using document root <none> and password file <none>
I20260928 06:26:08.656776 15166 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:08.656854 15166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:08.658351 15166 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:26:08.659020 15182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:08.659204 15166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:26:08.659322 15166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "66e6758f854d4098915bd11da97738f1"
format_stamp: "Formatted at 2026-09-28 06:26:06 on dist-test-slave-q03f"
server_key: "cdd40a976efb54ef18c392d358633cc3"
server_key_iv: "df2ec757c92ec52b4c9e5abdcb37d1f9"
server_key_version: "encryptionkey@0"
I20260928 06:26:08.659710 15166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:08.686031 15166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:08.688997 15166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:08.689211 15166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:08.689494 15166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:08.689869 15166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:08.689929 15166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:08.689993 15166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:08.690048 15166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:08.696988 15166 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:40967
I20260928 06:26:08.697034 15295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:40967 every 8 connection(s)
I20260928 06:26:08.697428 15166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:26:08.698652 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.698240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:56531)
0928 06:26:08.698406 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.698503 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
W20260928 06:26:08.699090 15296 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:35673 (0 consecutive failures): Network error: Failed to ping master at 127.31.214.126:35673: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
I20260928 06:26:08.699277 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 15166
I20260928 06:26:08.699417 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.699327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:50957)
0928 06:26:08.699362 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.699389 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 06:26:08.699638 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:38017
--local_ip_for_outbound_sockets=127.31.214.67
--tserver_master_addrs=127.31.214.126:35673
--webserver_port=41145
--webserver_interface=127.31.214.67
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:08.699700 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.699614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:50803)
0928 06:26:08.699658 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.699680 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:26:08.699806 15296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:26:08.786764 15299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:08.787027 15299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:08.787070 15299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:08.788264 15299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:08.788342 15299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:08.788389 15299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:26:08.789991 15299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:38017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=41145
--tserver_master_addrs=127.31.214.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:08.790304 15299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:08.790565 15299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:26:08.796137 15299 server_base.cc:1034] running on GCE node
W20260928 06:26:08.796196 15304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:08.796278 15307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:08.796139 15305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:08.796627 15299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:08.796912 15299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:08.798070 15299 hybrid_clock.cc:648] HybridClock initialized: now 1790576768798047 us; error 43 us; skew 500 ppm
Sep 28 06:26:08 dist-test-slave-q03f krb5kdc[13862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576768, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:08.800563 15299 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:26:08.801005 15299 webserver.cc:460] Webserver started at http://127.31.214.67:41145/ using document root <none> and password file <none>
I20260928 06:26:08.801221 15299 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:08.801280 15299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:08.802780 15299 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:08.803531 15314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:08.803774 15299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:08.803874 15299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "05ed5564fc464b39a61ac6a1a7523de8"
format_stamp: "Formatted at 2026-09-28 06:26:07 on dist-test-slave-q03f"
server_key: "09b6d2e6f76d26034fc9bedb89640d5a"
server_key_iv: "a6e8a269eb48c52d32b722ff063f6b84"
server_key_version: "encryptionkey@0"
I20260928 06:26:08.804242 15299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:08.817688 15299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:08.820253 15299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:08.820386 15299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:08.820598 15299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:08.820879 15299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:08.820910 15299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:08.820935 15299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:08.820947 15299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:08.827044 15299 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:38017
I20260928 06:26:08.827127 15427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:38017 every 8 connection(s)
I20260928 06:26:08.827494 15299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:26:08.828687 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.828264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:53575)
0928 06:26:08.828440 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.828544 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
W20260928 06:26:08.829152 15428 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:35673 (0 consecutive failures): Network error: Failed to ping master at 127.31.214.126:35673: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
I20260928 06:26:08.829473 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.829364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:54603)
0928 06:26:08.829416 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.829442 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:26:08.829777 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:08.829686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:43073)
0928 06:26:08.829731 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:08.829755 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:26:08.829896 15428 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:26:08.835274 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 15299
I20260928 06:26:09.608745 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:09.608474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:59325)
0928 06:26:09.608642 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:09.608692 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260928 06:26:09.700532 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:09.700280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:37309)
0928 06:26:09.700425 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:09.700465 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260928 06:26:09.715978 15434 raft_consensus.cc:482] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:09.716115 15434 raft_consensus.cc:504] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:09.716562 15434 leader_election.cc:288] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017), 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967)
Sep 28 06:26:09 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576768, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.66@KRBTEST.COM
Sep 28 06:26:09 dist-test-slave-q03f krb5kdc[13862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576768, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.67@KRBTEST.COM
I20260928 06:26:09.730559 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:09.716852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:40967 (local address 127.31.214.65:43359)
0928 06:26:09.717013 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:09.717022 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:09.717060 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:09.718196 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:09.718199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:09.718687 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:09.718971 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:09.718979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:09.720787 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:09.720788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:09.722186 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:09.722190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:09.723159 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:09.723160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:09.723384 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:09.724150 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:09.724154 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:09.726468 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:09.728886 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:09.728895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:09.728898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:09.729146 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:09.729427 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:09.729428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:09.729428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:09.729545 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:09.729987 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:09.729995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:09.730139 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 06:26:09.730245 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 06:26:09.730442 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":96,"thread_start_us":67,"threads_started":1}
I20260928 06:26:09.730571 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:09.716939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:43359 (local address 127.31.214.66:40967)
0928 06:26:09.717111 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:09.717126 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:09.717163 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:26:09.717743 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:09.717746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:09.717807 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:09.718001 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:09.719145 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:09.720636 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:09.722302 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:09.723039 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:09.723188 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:09.726616 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:09.726621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:09.726622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:09.726635 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:09.728773 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:09.729250 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:09.729253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:09.729254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:09.729344 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:09.729616 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:09.729617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:09.729618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:09.729882 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:09.730001 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:09.730315 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 06:26:09.730452 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260928 06:26:09.730873 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:09.716892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:43407 (local address 127.31.214.67:38017)
0928 06:26:09.717124 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:09.717132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:09.717163 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:26:09.717742 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:09.717745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:09.717824 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:09.718001 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:09.719193 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:09.720607 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:09.722298 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:09.723128 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:09.723246 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:09.726858 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:09.726862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:09.726862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:09.726874 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:09.729089 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:09.729525 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:09.729528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:09.729528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:09.729634 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:09.729895 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:09.729896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:09.729896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:09.730217 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:09.730320 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:09.730536 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:26:09.730691 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":47,"threads_started":1}
I20260928 06:26:09.731124 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:09.716769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:38017 (local address 127.31.214.65:43407)
0928 06:26:09.716856 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:09.716868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:09.716928 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:09.718196 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:09.718199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:09.718687 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:09.718971 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:09.718979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:09.720802 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:09.720803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:09.722189 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:09.722190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:09.723242 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:09.723243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:09.723384 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:09.724110 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:09.724123 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:09.726645 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:09.729213 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:09.729215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:09.729216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:09.729413 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:09.729713 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:09.729714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:09.729715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:09.729809 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:09.730301 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:09.730302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:09.730355 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:26:09.730451 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:26:09.730580 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":54}
W20260928 06:26:09.731559 15048 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
W20260928 06:26:09.732049 15051 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
I20260928 06:26:09.732113 15051 leader_election.cc:302] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14eca66fbead497ba67a704c3697bee2; no voters: 05ed5564fc464b39a61ac6a1a7523de8, 66e6758f854d4098915bd11da97738f1
I20260928 06:26:09.732251 15434 raft_consensus.cc:2726] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:26:09.830518 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:09.830265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:50393)
0928 06:26:09.830418 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:09.830466 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260928 06:26:10.609530 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:10.609273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:54681)
0928 06:26:10.609435 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:10.609481 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260928 06:26:10.701447 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:10.701153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:35517)
0928 06:26:10.701324 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:10.701371 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
I20260928 06:26:10.831298 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:10.831009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:56289)
0928 06:26:10.831196 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:10.831235 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260928 06:26:11.289288 15442 raft_consensus.cc:482] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:11.289402 15442 raft_consensus.cc:504] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:11.289634 15442 leader_election.cc:288] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017), 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967)
W20260928 06:26:11.290194 15051 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
W20260928 06:26:11.290277 15048 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
I20260928 06:26:11.290376 15048 leader_election.cc:302] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14eca66fbead497ba67a704c3697bee2; no voters: 05ed5564fc464b39a61ac6a1a7523de8, 66e6758f854d4098915bd11da97738f1
I20260928 06:26:11.290452 15442 raft_consensus.cc:2726] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:26:11.610503 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:11.610228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:35309)
0928 06:26:11.610408 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:11.610452 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":124,"threads_started":1}
I20260928 06:26:11.702270 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:11.702017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:35671)
0928 06:26:11.702179 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:11.702224 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260928 06:26:11.832187 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:11.831886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:59011)
0928 06:26:11.832072 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:11.832134 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":76,"threads_started":1}
I20260928 06:26:12.611393 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:12.611122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:51471)
0928 06:26:12.611296 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:12.611343 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260928 06:26:12.703125 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:12.702789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:49345)
0928 06:26:12.703020 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:12.703075 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260928 06:26:12.833078 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:12.832813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:41425)
0928 06:26:12.832984 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:12.833027 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260928 06:26:13.130486 15449 raft_consensus.cc:482] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:13.130616 15449 raft_consensus.cc:504] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:13.130848 15449 leader_election.cc:288] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017), 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967)
W20260928 06:26:13.131309 15051 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
W20260928 06:26:13.131397 15048 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
I20260928 06:26:13.131469 15048 leader_election.cc:302] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14eca66fbead497ba67a704c3697bee2; no voters: 05ed5564fc464b39a61ac6a1a7523de8, 66e6758f854d4098915bd11da97738f1
I20260928 06:26:13.131564 15449 raft_consensus.cc:2726] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:26:13.612331 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:13.612075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:41643)
0928 06:26:13.612230 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:13.612277 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260928 06:26:13.703923 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:13.703688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:37719)
0928 06:26:13.703837 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:13.703877 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260928 06:26:13.834090 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:13.833733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:50289)
0928 06:26:13.833915 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:13.834026 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260928 06:26:14.613180 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:14.612873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:39339)
0928 06:26:14.613077 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:14.613128 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260928 06:26:14.704694 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:14.704457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:50461)
0928 06:26:14.704609 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:14.704648 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260928 06:26:14.835027 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:14.834656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:59283)
0928 06:26:14.834863 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:14.834916 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
I20260928 06:26:15.613993 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:15.613692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:58053)
0928 06:26:15.613843 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:15.613895 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260928 06:26:15.705535 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:15.705263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:36007)
0928 06:26:15.705441 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:15.705486 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260928 06:26:15.835862 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:15.835611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:58137)
0928 06:26:15.835766 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:15.835811 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260928 06:26:16.614724 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:16.614481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:34629)
0928 06:26:16.614624 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:16.614670 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260928 06:26:16.706737 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:16.706430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:49181)
0928 06:26:16.706594 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:16.706668 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260928 06:26:16.836717 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:16.836489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:56457)
0928 06:26:16.836624 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:16.836663 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260928 06:26:16.874454 15462 raft_consensus.cc:482] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:16.874568 15462 raft_consensus.cc:504] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:16.874816 15462 leader_election.cc:288] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017), 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967)
W20260928 06:26:16.875320 15051 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
W20260928 06:26:16.875424 15048 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
I20260928 06:26:16.875464 15051 leader_election.cc:302] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14eca66fbead497ba67a704c3697bee2; no voters: 05ed5564fc464b39a61ac6a1a7523de8, 66e6758f854d4098915bd11da97738f1
I20260928 06:26:16.875566 15462 raft_consensus.cc:2726] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:26:17.615530 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:17.615287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:53025)
0928 06:26:17.615434 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:17.615480 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260928 06:26:17.707594 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:17.707365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:43083)
0928 06:26:17.707507 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:17.707544 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 06:26:17.837509 15465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:17.837232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:50225)
0928 06:26:17.837399 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:17.837443 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260928 06:26:18.393931 15466 raft_consensus.cc:482] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:18.394071 15466 raft_consensus.cc:504] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ed5564fc464b39a61ac6a1a7523de8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38017 } } peers { permanent_uuid: "66e6758f854d4098915bd11da97738f1" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 40967 } } peers { permanent_uuid: "14eca66fbead497ba67a704c3697bee2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 37417 } }
I20260928 06:26:18.394320 15466 leader_election.cc:288] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017), 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967)
W20260928 06:26:18.394789 15048 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66e6758f854d4098915bd11da97738f1 (127.31.214.66:40967): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
W20260928 06:26:18.394902 15051 leader_election.cc:341] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05ed5564fc464b39a61ac6a1a7523de8 (127.31.214.67:38017): Not found: Tablet not found: f9a614f1d2064227800af942deb2773b
I20260928 06:26:18.395046 15051 leader_election.cc:302] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14eca66fbead497ba67a704c3697bee2; no voters: 05ed5564fc464b39a61ac6a1a7523de8, 66e6758f854d4098915bd11da97738f1
I20260928 06:26:18.395133 15466 raft_consensus.cc:2726] T f9a614f1d2064227800af942deb2773b P 14eca66fbead497ba67a704c3697bee2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:26:18.616523 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:18.616196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:49235)
0928 06:26:18.616410 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:18.616468 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
I20260928 06:26:18.708398 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:18.708103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:50241)
0928 06:26:18.708293 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:18.708350 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":137,"threads_started":1}
I20260928 06:26:18.838505 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:18.838247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:39745)
0928 06:26:18.838408 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:18.838454 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260928 06:26:19.617488 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:19.617214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:41863)
0928 06:26:19.617379 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:19.617433 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260928 06:26:19.709283 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:19.709046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:54319)
0928 06:26:19.709193 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:19.709236 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260928 06:26:19.839476 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:19.839172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:41945)
0928 06:26:19.839373 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:19.839424 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":107,"threads_started":1}
I20260928 06:26:20.618528 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:20.618199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:60949)
0928 06:26:20.618401 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:20.618466 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":147,"threads_started":1}
I20260928 06:26:20.710320 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:20.710061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:34039)
0928 06:26:20.710224 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:20.710271 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260928 06:26:20.840466 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:20.840160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:50073)
0928 06:26:20.840354 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:20.840409 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
I20260928 06:26:21.619381 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:21.619099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:47937)
0928 06:26:21.619279 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:21.619329 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260928 06:26:21.711086 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:21.710864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:53285)
0928 06:26:21.710996 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:21.711037 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260928 06:26:21.841513 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:21.841213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:39015)
0928 06:26:21.841383 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:21.841460 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260928 06:26:22.620312 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:22.620051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:58503)
0928 06:26:22.620218 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:22.620258 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260928 06:26:22.712040 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:22.711709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:38039)
0928 06:26:22.711945 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:22.711990 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":132,"threads_started":1}
I20260928 06:26:22.842474 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:22.842198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:60735)
0928 06:26:22.842362 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:22.842422 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260928 06:26:23.621419 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:23.621158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.65:44205)
0928 06:26:23.621320 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:23.621371 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260928 06:26:23.713086 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:23.712709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.66:48525)
0928 06:26:23.712963 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:23.713034 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":82,"threads_started":1}
I20260928 06:26:23.837340 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 15003
I20260928 06:26:23.842835 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 15166
I20260928 06:26:23.844872 15484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:23.844610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35673 (local address 127.31.214.67:47483)
0928 06:26:23.844762 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:23.844804 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260928 06:26:23.847791 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 15299
2026-09-28T06:26:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18619 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:26:23 dist-test-slave-q03f krb5kdc[15487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:26:23 dist-test-slave-q03f krb5kdc[15487](info): set up 2 sockets
Sep 28 06:26:23 dist-test-slave-q03f krb5kdc[15487](info): commencing operation
krb5kdc: starting...
W20260928 06:26:24.938289 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:26:25 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:26:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:26:25.078653 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45905
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:45905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:25.167356 15500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:25.167599 15500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:25.167624 15500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:25.168964 15500 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:26:25.169026 15500 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:26:25.169040 15500 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:26:25.169052 15500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:26:25.169067 15500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:25.169080 15500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:25.169092 15500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:26:25.169162 15500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:26:25.170938 15500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:45905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:25.171263 15500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:25.171543 15500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:26:25.177405 15500 server_base.cc:1034] running on GCE node
W20260928 06:26:25.177539 15508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:25.177505 15506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:25.177498 15505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:25.178078 15500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:25.178445 15500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:25.179648 15500 hybrid_clock.cc:648] HybridClock initialized: now 1790576785179637 us; error 46 us; skew 500 ppm
Sep 28 06:26:25 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:25.182708 15500 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:26:25.183172 15500 webserver.cc:460] Webserver started at http://127.31.214.126:34895/ using document root <none> and password file <none>
I20260928 06:26:25.183406 15500 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:25.183454 15500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:25.183552 15500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:25.184718 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "fdc241aa319e41c99aa4c5b2df91fd4e"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "56eb71b49f28e6995a968bfd3ef9dde5"
server_key_iv: "5fc23339cedbb506f8aab6834a14b1ec"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.185055 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "fdc241aa319e41c99aa4c5b2df91fd4e"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "56eb71b49f28e6995a968bfd3ef9dde5"
server_key_iv: "5fc23339cedbb506f8aab6834a14b1ec"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.186846 15500 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260928 06:26:25.187726 15515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.188079 15500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:25.188160 15500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "fdc241aa319e41c99aa4c5b2df91fd4e"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "56eb71b49f28e6995a968bfd3ef9dde5"
server_key_iv: "5fc23339cedbb506f8aab6834a14b1ec"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.188267 15500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:25.205041 15500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:25.207589 15500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:25.207734 15500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:25.211925 15500 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:45905
I20260928 06:26:25.211990 15567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:45905 every 8 connection(s)
I20260928 06:26:25.212347 15500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:26:25.213425 15568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:25.214787 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 15500
I20260928 06:26:25.214908 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:26:25.215134 32601 external_mini_cluster.cc:1428] Setting key 7cc15b9eb502ccb370bca1d714d3f7cf
I20260928 06:26:25.216531 15568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Bootstrap starting.
I20260928 06:26:25.217196 15568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:25.217514 15568 log.cc:824] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:25.218266 15568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: No bootstrap required, opened a new log
I20260928 06:26:25.220202 15568 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:26:25.220408 15568 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:25.220448 15568 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc241aa319e41c99aa4c5b2df91fd4e, State: Initialized, Role: FOLLOWER
I20260928 06:26:25.220587 15568 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:26:25.220712 15568 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:26:25.220764 15568 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:26:25.220837 15568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:25.221441 15568 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:26:25.221560 15568 leader_election.cc:302] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc241aa319e41c99aa4c5b2df91fd4e; no voters:
I20260928 06:26:25.221764 15568 leader_election.cc:288] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:26:25.221803 15573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:26:25.222024 15573 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 1 LEADER]: Becoming Leader. State: Replica: fdc241aa319e41c99aa4c5b2df91fd4e, State: Running, Role: LEADER
I20260928 06:26:25.222105 15568 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:26:25.222200 15573 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
Sep 28 06:26:25 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:25.222609 15573 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdc241aa319e41c99aa4c5b2df91fd4e. Latest consensus state: current_term: 1 leader_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } } }
I20260928 06:26:25.222775 15573 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:25.223069 15582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:26:25.223549 15574 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } } }
I20260928 06:26:25.224012 15574 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:25.223968 15582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:26:25.226276 15582 catalog_manager.cc:1347] Generated new cluster ID: 5f325166f3784fa0b39b9dd8b86d3bb5
I20260928 06:26:25.226331 15582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:26:25.227440 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.215777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.31.214.126:45905)
0928 06:26:25.216017 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.216025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.216052 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:26:25.216470 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.216477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.216526 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.216795 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:25.217536 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.218631 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.219447 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.219889 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.220051 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.222963 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:25.222968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:25.222969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:25.222984 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:25.225185 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.225664 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.225667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.225668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.225787 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.226059 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.226060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.226060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.226391 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:25.226550 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.227058 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.227210 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260928 06:26:25.237581 15582 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:26:25.238356 15582 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:26:25.244833 15582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Generated new TSK 0
I20260928 06:26:25.245136 15582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:26:25.278429 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45905
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:25.369094 15594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:25.369377 15594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:25.369421 15594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:25.370692 15594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:25.370771 15594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:25.370818 15594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:26:25.372455 15594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:25.372785 15594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:25.373100 15594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:25.378907 15599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:25.378921 15602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:25.378921 15600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:25.378938 15594 server_base.cc:1034] running on GCE node
I20260928 06:26:25.379412 15594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:25.379685 15594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:25.380883 15594 hybrid_clock.cc:648] HybridClock initialized: now 1790576785380867 us; error 66 us; skew 500 ppm
Sep 28 06:26:25 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:25.384016 15594 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:26:25.384552 15594 webserver.cc:460] Webserver started at http://127.31.214.65:36975/ using document root <none> and password file <none>
I20260928 06:26:25.384766 15594 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:25.384811 15594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:25.384909 15594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:25.386157 15594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "60f5f2132f3c49bda8af16c8f9c6e7db"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "4b69b1742b333137bfeef26103dbe9ed"
server_key_iv: "20295fb2295c04a2638ff1c205e728bd"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.386507 15594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "60f5f2132f3c49bda8af16c8f9c6e7db"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "4b69b1742b333137bfeef26103dbe9ed"
server_key_iv: "20295fb2295c04a2638ff1c205e728bd"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.387995 15594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:26:25.388906 15609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.389202 15594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:26:25.389305 15594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "60f5f2132f3c49bda8af16c8f9c6e7db"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "4b69b1742b333137bfeef26103dbe9ed"
server_key_iv: "20295fb2295c04a2638ff1c205e728bd"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.389374 15594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:25.405992 15594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:25.409363 15594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:25.409564 15594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:25.409859 15594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:25.410326 15594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:25.410394 15594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.410439 15594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:25.410466 15594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.419289 15594 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:32935
I20260928 06:26:25.419335 15722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:32935 every 8 connection(s)
I20260928 06:26:25.419835 15594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:26:25.425252 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 15594
I20260928 06:26:25.425437 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:26:25.425793 32601 external_mini_cluster.cc:1428] Setting key 61439b5e01191b1d95c4d84b29f1c3c7
Sep 28 06:26:25 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:25.433573 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.420803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:58865 (local address 127.31.214.126:45905)
0928 06:26:25.420873 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.420875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.421220 (+ 345us) server_negotiation.cc:407] Connection header received
0928 06:26:25.422198 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.422200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.422217 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.422296 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:25.423846 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.424686 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.426592 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.426923 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.427013 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.430102 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:25.430104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:25.430105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:25.430114 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:25.431966 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.432427 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.432428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.432428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.432502 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.432797 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.432798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.432799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.432939 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:25.433020 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.433357 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.433469 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:26:25.433858 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.420804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:58865)
0928 06:26:25.421023 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:25.421087 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:25.422027 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:25.422395 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:25.422400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:25.423384 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:25.423722 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.423729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.424797 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.424799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.426486 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.426490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.427022 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.427023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.427185 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.427884 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:25.427894 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:25.430006 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:25.432063 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:25.432073 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:25.432076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:25.432330 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:25.432575 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:25.432576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:25.432577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:25.432711 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:25.433023 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:25.433032 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:25.433196 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 06:26:25.433310 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:26:25.433507 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260928 06:26:25.434458 15723 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45905
I20260928 06:26:25.434568 15723 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:25.434813 15723 heartbeater.cc:511] Master 127.31.214.126:45905 requested a full tablet report, sending...
I20260928 06:26:25.435510 15532 ts_manager.cc:194] Registered new tserver with Master: 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65:32935)
I20260928 06:26:25.436283 15532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:58865
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:26:25.462815 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45905
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:25.549213 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:25.549471 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:25.549496 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:25.550741 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:25.550796 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:25.550824 15728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:26:25.552328 15728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:25.552597 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:25.552851 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:26:25.558372 15728 server_base.cc:1034] running on GCE node
W20260928 06:26:25.558316 15734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:25.558378 15733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:25.558336 15736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:25.558795 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:25.559088 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:25.560271 15728 hybrid_clock.cc:648] HybridClock initialized: now 1790576785560257 us; error 42 us; skew 500 ppm
Sep 28 06:26:25 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:25.563355 15728 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:26:25.563889 15728 webserver.cc:460] Webserver started at http://127.31.214.66:36995/ using document root <none> and password file <none>
I20260928 06:26:25.564152 15728 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:25.564230 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:25.564396 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:25.565688 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "d73ad994ff5b49cf90873cd7c692d81a"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "aa11eef1ed65ffe38920b879c1741ae7"
server_key_iv: "73f1e1741ee6de3fab0b105127e85d7b"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.566159 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "d73ad994ff5b49cf90873cd7c692d81a"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "aa11eef1ed65ffe38920b879c1741ae7"
server_key_iv: "73f1e1741ee6de3fab0b105127e85d7b"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.567790 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:25.568758 15743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.569032 15728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:25.569159 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "d73ad994ff5b49cf90873cd7c692d81a"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "aa11eef1ed65ffe38920b879c1741ae7"
server_key_iv: "73f1e1741ee6de3fab0b105127e85d7b"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.569247 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:25.588876 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:25.591620 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:25.591841 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:25.592197 15728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:25.592579 15728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:25.592648 15728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.592698 15728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:25.592729 15728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.600466 15728 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:33637
I20260928 06:26:25.600517 15856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:33637 every 8 connection(s)
I20260928 06:26:25.600888 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:26:25 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:25.609208 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 15728
I20260928 06:26:25.609337 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:26:25.609580 32601 external_mini_cluster.cc:1428] Setting key 803bc4dbc74fd5c9a30a9253eb5e30cd
I20260928 06:26:25.611593 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.601653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:60347)
0928 06:26:25.601918 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:25.601995 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:25.602546 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:25.602806 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:25.602811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:25.603485 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:25.603688 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.603693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.604467 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.604469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.605289 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.605291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.605677 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.605678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.605807 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.606400 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:25.606409 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:25.607935 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:25.609812 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:25.609821 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:25.609823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:25.610194 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:25.610451 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:25.610452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:25.610452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:25.610607 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:25.611009 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:25.611016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:25.611154 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:26:25.611248 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:26:25.611398 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
I20260928 06:26:25.611598 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.601740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:60347 (local address 127.31.214.126:45905)
0928 06:26:25.601890 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.601892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.602091 (+ 199us) server_negotiation.cc:407] Connection header received
0928 06:26:25.602655 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.602656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.602669 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.602726 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:25.603794 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.604305 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.605386 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.605584 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.605657 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.608035 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:25.608036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:25.608037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:25.608045 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:25.609735 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.610273 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.610275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.610275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.610370 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.610725 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.610726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.610726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.610907 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:25.610993 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.611329 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.611439 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 06:26:25.612830 15857 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45905
I20260928 06:26:25.612934 15857 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:25.613222 15857 heartbeater.cc:511] Master 127.31.214.126:45905 requested a full tablet report, sending...
I20260928 06:26:25.613653 15532 ts_manager.cc:194] Registered new tserver with Master: d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637)
I20260928 06:26:25.614212 15532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:60347
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:26:25.641577 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45905
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:25.727487 15862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:25.727721 15862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:25.727751 15862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:25.728925 15862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:25.728963 15862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:25.728991 15862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:26:25.730525 15862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:25.730727 15862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:25.730917 15862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:25.737129 15870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:25.737123 15867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:25.737125 15868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:25.737145 15862 server_base.cc:1034] running on GCE node
I20260928 06:26:25.737555 15862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:25.737838 15862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:25.739033 15862 hybrid_clock.cc:648] HybridClock initialized: now 1790576785739014 us; error 42 us; skew 500 ppm
Sep 28 06:26:25 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:25.742257 15862 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:26:25.742909 15862 webserver.cc:460] Webserver started at http://127.31.214.67:34833/ using document root <none> and password file <none>
I20260928 06:26:25.743212 15862 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:25.743273 15862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:25.743438 15862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:26:25.745201 15862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "c57e4523cdb7bf5eb69b12601615207a"
server_key_iv: "d93183ca2ba0242aa8a73b23d48370f6"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.745741 15862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "c57e4523cdb7bf5eb69b12601615207a"
server_key_iv: "d93183ca2ba0242aa8a73b23d48370f6"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.747502 15862 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:26:25.748394 15877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.748622 15862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:25.748742 15862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "c57e4523cdb7bf5eb69b12601615207a"
server_key_iv: "d93183ca2ba0242aa8a73b23d48370f6"
server_key_version: "encryptionkey@0"
I20260928 06:26:25.748852 15862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:25.764043 15862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:25.766602 15862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:25.766796 15862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:25.767092 15862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:25.767505 15862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:25.767561 15862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.767606 15862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:25.767632 15862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:25.774897 15862 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:37873
I20260928 06:26:25.775143 15990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:37873 every 8 connection(s)
I20260928 06:26:25.775363 15862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:26:25.777348 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 15862
I20260928 06:26:25.777428 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:26:25.777622 32601 external_mini_cluster.cc:1428] Setting key ef546f09e79d95749cb1384a3c3f0a50
Sep 28 06:26:25 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:25.787843 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.776290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:57929 (local address 127.31.214.126:45905)
0928 06:26:25.776347 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.776349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.776604 (+ 255us) server_negotiation.cc:407] Connection header received
0928 06:26:25.777802 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.777804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.777816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.777899 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:25.779317 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.779857 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.780939 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.781163 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.781243 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.784336 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:25.784338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:25.784338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:25.784346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:25.785933 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.786620 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.786622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.786622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.786690 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.787041 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.787041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.787042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.787174 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:25.787238 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.787660 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.787743 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:26:25.788283 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.776271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:57929)
0928 06:26:25.776464 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:25.776512 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:25.777094 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:25.778156 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:25.778161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:25.778961 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:25.779174 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.779180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.779963 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.779964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.780840 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.780843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.781259 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.781260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.781403 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.782052 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:25.782061 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:25.784187 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:25.786068 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:25.786077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:25.786079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:25.786505 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:25.786779 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:25.786780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:25.786781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:25.786959 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:25.787251 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:25.787259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:25.787486 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 06:26:25.787663 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 06:26:25.787928 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
I20260928 06:26:25.788978 15991 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45905
I20260928 06:26:25.789113 15991 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:25.789366 15991 heartbeater.cc:511] Master 127.31.214.126:45905 requested a full tablet report, sending...
I20260928 06:26:25.789781 15532 ts_manager.cc:194] Registered new tserver with Master: 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873)
I20260928 06:26:25.790500 15532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:57929
I20260928 06:26:25.801007 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:26:25.810606 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.802989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.31.214.126:45905)
0928 06:26:25.803067 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.803070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.803296 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:26:25.803376 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.803377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.803389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.803453 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:25.804077 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.804667 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.805515 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.805740 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.805837 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.807282 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:25.807283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:25.807284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:25.807292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:25.809001 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.809492 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.809494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.809494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.809568 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:25.809826 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:25.809827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:25.809827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:25.810097 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:25.810221 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.810387 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.810489 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:26:25.812861 15532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:26:25.816183 15532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:26:25.824739 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.819264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:37873 (local address 127.0.0.1:60986)
0928 06:26:25.819870 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:25.819913 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:25.819972 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:25.820333 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:25.820337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:25.820342 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:25.820439 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.820441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.821820 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.821821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.823240 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.823243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.824326 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.824328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.824462 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.824478 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:26:25.824557 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:26:25.824635 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":30,"thread_start_us":86,"threads_started":1}
I20260928 06:26:25.824755 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.819599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.31.214.65:32935)
0928 06:26:25.820613 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.820618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.820630 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:26:25.820671 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.820673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.820691 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.820777 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:25.821004 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.822091 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.823413 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.824133 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.824225 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.824282 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.824622 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.824665 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":109,"threads_started":1}
I20260928 06:26:25.825235 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.819556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.31.214.67:37873)
0928 06:26:25.819770 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.819776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.820072 (+ 296us) server_negotiation.cc:407] Connection header received
0928 06:26:25.820124 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.820128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.820146 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.820227 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:25.820540 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.821673 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.823356 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.824132 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.824213 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.824282 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.825074 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.825122 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260928 06:26:25.824996 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.819264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:32935 (local address 127.0.0.1:57770)
0928 06:26:25.819964 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:25.819978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:25.820401 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:25.820873 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:25.820874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:25.820877 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:25.820918 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.820920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.822228 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.822229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.823314 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.823316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.824329 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.824330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.824468 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.824480 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 06:26:25.824557 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:26:25.824930 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":381,"thread_start_us":145,"threads_started":1}
I20260928 06:26:25.826036 15925 tablet_service.cc:1467] Processing CreateTablet for tablet 07e4c35477dc41ab98c05b04ad698268 (DEFAULT_TABLE table=test-table [id=7b9c2c28cfdf41e7a2bb779904b19c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:25.826277 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.819259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33637 (local address 127.0.0.1:43510)
0928 06:26:25.819840 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:25.819888 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:25.819972 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:25.820333 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:25.820337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:25.820339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:25.820439 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.820442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.822541 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.822542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.824146 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.824149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.825931 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.825933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.826041 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.826048 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:26:25.826085 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 06:26:25.826142 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":7,"thread_start_us":95,"threads_started":1}
I20260928 06:26:25.826488 15925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e4c35477dc41ab98c05b04ad698268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:25.826799 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.819557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.31.214.66:33637)
0928 06:26:25.819779 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.819783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.820072 (+ 289us) server_negotiation.cc:407] Connection header received
0928 06:26:25.820133 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.820135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.820151 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.820233 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:25.820518 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.822384 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.824315 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.825812 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.826494 (+ 682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.826577 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.826607 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.826666 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
I20260928 06:26:25.827522 15783 tablet_service.cc:1467] Processing CreateTablet for tablet 07e4c35477dc41ab98c05b04ad698268 (DEFAULT_TABLE table=test-table [id=7b9c2c28cfdf41e7a2bb779904b19c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:25.827903 15783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e4c35477dc41ab98c05b04ad698268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:25.830296 16008 tablet_bootstrap.cc:492] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Bootstrap starting.
I20260928 06:26:25.830826 15657 tablet_service.cc:1467] Processing CreateTablet for tablet 07e4c35477dc41ab98c05b04ad698268 (DEFAULT_TABLE table=test-table [id=7b9c2c28cfdf41e7a2bb779904b19c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:25.831229 15657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e4c35477dc41ab98c05b04ad698268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:25.831944 16008 tablet_bootstrap.cc:654] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:25.832808 16008 log.cc:824] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:25.833840 16008 tablet_bootstrap.cc:492] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: No bootstrap required, opened a new log
I20260928 06:26:25.833921 16008 ts_tablet_manager.cc:1397] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:26:25.834566 16011 tablet_bootstrap.cc:492] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Bootstrap starting.
I20260928 06:26:25.835026 16009 tablet_bootstrap.cc:492] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: Bootstrap starting.
I20260928 06:26:25.835510 16011 tablet_bootstrap.cc:654] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:25.835794 16009 tablet_bootstrap.cc:654] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:25.836313 16011 log.cc:824] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:25.836408 16009 log.cc:824] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:25.837080 16008 raft_consensus.cc:348] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:25.837241 16011 tablet_bootstrap.cc:492] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: No bootstrap required, opened a new log
I20260928 06:26:25.837301 16008 raft_consensus.cc:374] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:25.837348 16008 raft_consensus.cc:729] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c5ff5c67b74ad9b3bc876ad72c15e8, State: Initialized, Role: FOLLOWER
I20260928 06:26:25.837469 16009 tablet_bootstrap.cc:492] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: No bootstrap required, opened a new log
I20260928 06:26:25.837544 16009 ts_tablet_manager.cc:1397] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:26:25.837505 16011 ts_tablet_manager.cc:1397] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:26:25.837564 16008 consensus_queue.cc:260] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:25.837922 16008 ts_tablet_manager.cc:1428] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 06:26:25.838088 15991 heartbeater.cc:503] Master 127.31.214.126:45905 was elected leader, sending a full tablet report...
I20260928 06:26:25.839823 16009 raft_consensus.cc:348] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:25.840040 16009 raft_consensus.cc:374] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:25.840092 16009 raft_consensus.cc:729] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73ad994ff5b49cf90873cd7c692d81a, State: Initialized, Role: FOLLOWER
I20260928 06:26:25.839941 16011 raft_consensus.cc:348] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:25.840289 16009 consensus_queue.cc:260] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:25.840378 16011 raft_consensus.cc:374] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:25.840528 16011 raft_consensus.cc:729] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f5f2132f3c49bda8af16c8f9c6e7db, State: Initialized, Role: FOLLOWER
I20260928 06:26:25.840654 16009 ts_tablet_manager.cc:1428] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:26:25.840752 15857 heartbeater.cc:503] Master 127.31.214.126:45905 was elected leader, sending a full tablet report...
I20260928 06:26:25.840790 16011 consensus_queue.cc:260] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:25.841253 16011 ts_tablet_manager.cc:1428] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:26:25.841425 15723 heartbeater.cc:503] Master 127.31.214.126:45905 was elected leader, sending a full tablet report...
I20260928 06:26:25.845160 16016 raft_consensus.cc:482] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:25.845261 16016 raft_consensus.cc:504] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:25.845819 16016 leader_election.cc:288] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873), d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637)
I20260928 06:26:25.849633 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.846061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33637 (local address 127.31.214.65:60809)
0928 06:26:25.846127 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:25.846143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:25.846211 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:25.846386 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:25.846387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:25.846389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:25.846482 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.846484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.847593 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.847594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.848629 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.848632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.849283 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.849285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.849430 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.849435 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:26:25.849497 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:26:25.849546 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:26:25.849659 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.846185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:60809 (local address 127.31.214.66:33637)
0928 06:26:25.846235 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.846238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.846267 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:26:25.846297 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.846298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.846308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.846372 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:25.846560 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.847484 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.848732 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.849192 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.849257 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.849265 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.849513 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.849566 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:26:25.849800 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.846141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:52757 (local address 127.31.214.67:37873)
0928 06:26:25.846196 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.846198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.846476 (+ 278us) server_negotiation.cc:407] Connection header received
0928 06:26:25.846556 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.846557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.846567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.846629 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:25.846758 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.847559 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.848805 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.849370 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.849459 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.849469 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.849657 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.849709 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:26:25.849836 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.846126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:37873 (local address 127.31.214.65:52757)
0928 06:26:25.846386 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:25.846399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:25.846465 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:25.846635 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:25.846635 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:25.846637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:25.846694 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.846695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.847633 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.847635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.848660 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:25.848663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.849466 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.849467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:25.849567 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.849572 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:26:25.849637 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:26:25.849685 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":3,"thread_start_us":140,"threads_started":1}
I20260928 06:26:25.850229 15945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e4c35477dc41ab98c05b04ad698268" candidate_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" is_pre_election: true
I20260928 06:26:25.850257 15811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e4c35477dc41ab98c05b04ad698268" candidate_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73ad994ff5b49cf90873cd7c692d81a" is_pre_election: true
I20260928 06:26:25.850449 15945 raft_consensus.cc:2454] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60f5f2132f3c49bda8af16c8f9c6e7db in term 0.
I20260928 06:26:25.850458 15811 raft_consensus.cc:2454] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60f5f2132f3c49bda8af16c8f9c6e7db in term 0.
I20260928 06:26:25.850709 15610 leader_election.cc:302] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c5ff5c67b74ad9b3bc876ad72c15e8, 60f5f2132f3c49bda8af16c8f9c6e7db; no voters:
I20260928 06:26:25.850871 16016 raft_consensus.cc:2781] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:26:25.850955 16016 raft_consensus.cc:482] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:26:25.850996 16016 raft_consensus.cc:3037] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Advancing to term 1
W20260928 06:26:25.851500 15858 tablet.cc:2369] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:26:25.852063 16016 raft_consensus.cc:504] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:25.852232 16016 leader_election.cc:288] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 1 election: Requested vote from peers 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873), d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637)
I20260928 06:26:25.852499 15945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e4c35477dc41ab98c05b04ad698268" candidate_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8"
I20260928 06:26:25.852502 15811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e4c35477dc41ab98c05b04ad698268" candidate_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73ad994ff5b49cf90873cd7c692d81a"
I20260928 06:26:25.852586 15811 raft_consensus.cc:3037] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:25.852586 15945 raft_consensus.cc:3037] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:25.853578 15811 raft_consensus.cc:2454] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60f5f2132f3c49bda8af16c8f9c6e7db in term 1.
I20260928 06:26:25.853578 15945 raft_consensus.cc:2454] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60f5f2132f3c49bda8af16c8f9c6e7db in term 1.
I20260928 06:26:25.853801 15611 leader_election.cc:302] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f5f2132f3c49bda8af16c8f9c6e7db, d73ad994ff5b49cf90873cd7c692d81a; no voters:
I20260928 06:26:25.854000 16016 raft_consensus.cc:2781] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:26:25.854206 16016 raft_consensus.cc:686] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 LEADER]: Becoming Leader. State: Replica: 60f5f2132f3c49bda8af16c8f9c6e7db, State: Running, Role: LEADER
I20260928 06:26:25.854308 16016 consensus_queue.cc:237] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:25.855297 15531 catalog_manager.cc:5495] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db reported cstate change: term changed from 0 to 1, leader changed from <none> to 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65). New cstate: current_term: 1 leader_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20260928 06:26:25.870874 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.868346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.31.214.65:32935)
0928 06:26:25.868416 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:25.868421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:25.868433 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:26:25.868468 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:25.868469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:25.868489 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:25.868568 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:26:25.868696 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.869214 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.870047 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:25.870255 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:25.870333 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:25.870521 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:26:25.870632 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:26:25.870727 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:25.870746 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:26:25.870769 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:26:25.872990 15945 raft_consensus.cc:1261] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Refusing update from remote peer 60f5f2132f3c49bda8af16c8f9c6e7db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:25.873040 15811 raft_consensus.cc:1261] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [term 1 FOLLOWER]: Refusing update from remote peer 60f5f2132f3c49bda8af16c8f9c6e7db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:25.873409 16016 consensus_queue.cc:1035] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:25.873591 16018 consensus_queue.cc:1035] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:25.876204 16021 mvcc.cc:205] Tried to move back new op lower bound from 7334202514932453376 to 7334202514860072960. Current Snapshot: MvccSnapshot[applied={T|T < 7334202514932453376}]
I20260928 06:26:25.876603 16023 mvcc.cc:205] Tried to move back new op lower bound from 7334202514932453376 to 7334202514860072960. Current Snapshot: MvccSnapshot[applied={T|T < 7334202514932453376}]
I20260928 06:26:25.876425 16022 mvcc.cc:205] Tried to move back new op lower bound from 7334202514932453376 to 7334202514860072960. Current Snapshot: MvccSnapshot[applied={T|T < 7334202514932453376}]
I20260928 06:26:25.879441 15532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:26:25.879530 15532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:26:25.880348 15532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Sending DeleteTablet for 3 replicas of tablet 07e4c35477dc41ab98c05b04ad698268
I20260928 06:26:25.880726 15657 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e4c35477dc41ab98c05b04ad698268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:25 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:57770
I20260928 06:26:25.880761 15783 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e4c35477dc41ab98c05b04ad698268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:25 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:43510
I20260928 06:26:25.880751 15925 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e4c35477dc41ab98c05b04ad698268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:25 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:60986
I20260928 06:26:25.880910 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 15594
I20260928 06:26:25.881513 16032 tablet_replica.cc:330] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: stopping tablet replica
I20260928 06:26:25.881642 16032 raft_consensus.cc:2229] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:26:25.881763 16032 raft_consensus.cc:2258] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:25.882066 16033 tablet_replica.cc:330] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: stopping tablet replica
I20260928 06:26:25.882185 16033 raft_consensus.cc:2229] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:26:25.882297 16033 raft_consensus.cc:2258] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:25.882296 16032 ts_tablet_manager.cc:1905] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:25.883858 16033 ts_tablet_manager.cc:1905] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:25.884209 16032 ts_tablet_manager.cc:1918] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:25.884310 16032 log.cc:1196] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/07e4c35477dc41ab98c05b04ad698268
I20260928 06:26:25.884605 16032 ts_tablet_manager.cc:1939] T 07e4c35477dc41ab98c05b04ad698268 P d73ad994ff5b49cf90873cd7c692d81a: Deleting consensus metadata
I20260928 06:26:25.885105 15517 catalog_manager.cc:4841] TS d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637): tablet 07e4c35477dc41ab98c05b04ad698268 (table test-table [id=7b9c2c28cfdf41e7a2bb779904b19c15]) successfully deleted
I20260928 06:26:25.886068 16033 ts_tablet_manager.cc:1918] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:25.886145 16033 log.cc:1196] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/07e4c35477dc41ab98c05b04ad698268
I20260928 06:26:25.886400 16033 ts_tablet_manager.cc:1939] T 07e4c35477dc41ab98c05b04ad698268 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Deleting consensus metadata
I20260928 06:26:25.887076 15516 catalog_manager.cc:4841] TS 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873): tablet 07e4c35477dc41ab98c05b04ad698268 (table test-table [id=7b9c2c28cfdf41e7a2bb779904b19c15]) successfully deleted
W20260928 06:26:25.887929 15519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:32935 (error 108)
I20260928 06:26:25.888048 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 15728
I20260928 06:26:25.888993 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:25.888845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:32935 (local address 127.0.0.1:57784)
0928 06:26:25.888911 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:25.888954 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 06:26:25.889133 15519 catalog_manager.cc:4584] TS 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65:32935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07e4c35477dc41ab98c05b04ad698268: Network error: Client connection negotiation failed: client connection to 127.31.214.65:32935: connect: Connection refused (error 111)
I20260928 06:26:25.894150 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 15862
I20260928 06:26:25.899153 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 15500
I20260928 06:26:45.905752 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45905
--webserver_interface=127.31.214.126
--webserver_port=34895
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:45905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:26:45.989751 16035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:45.990057 16035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:45.990098 16035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:45.991303 16035 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:26:45.991374 16035 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:26:45.991394 16035 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:26:45.991412 16035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:26:45.991441 16035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:45.991472 16035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:45.991493 16035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:26:45.991515 16035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:26:45.993127 16035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:45905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:45.993431 16035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:45.993680 16035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:46.004503 16041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:46.004503 16044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:46.004729 16035 server_base.cc:1034] running on GCE node
W20260928 06:26:46.004832 16042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:46.005246 16035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:46.005676 16035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:46.006846 16035 hybrid_clock.cc:648] HybridClock initialized: now 1790576806006807 us; error 54 us; skew 500 ppm
Sep 28 06:26:46 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:46.010454 16035 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:26:46.011071 16035 webserver.cc:460] Webserver started at http://127.31.214.126:34895/ using document root <none> and password file <none>
I20260928 06:26:46.011426 16035 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:46.011559 16035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:46.013154 16035 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:46.013916 16051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:46.014142 16035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:46.014292 16035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "fdc241aa319e41c99aa4c5b2df91fd4e"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "56eb71b49f28e6995a968bfd3ef9dde5"
server_key_iv: "5fc23339cedbb506f8aab6834a14b1ec"
server_key_version: "encryptionkey@0"
I20260928 06:26:46.014744 16035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:46.056371 16035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:46.059194 16035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:46.059386 16035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:46.063992 16035 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:45905
I20260928 06:26:46.064070 16103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:45905 every 8 connection(s)
I20260928 06:26:46.064508 16035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:26:46.066399 16104 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:26:46.067267 16104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Bootstrap starting.
I20260928 06:26:46.072294 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 16035
I20260928 06:26:46.072636 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:32935
--local_ip_for_outbound_sockets=127.31.214.65
--tserver_master_addrs=127.31.214.126:45905
--webserver_port=36975
--webserver_interface=127.31.214.65
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:46.077333 16104 log.cc:824] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:46.080763 16104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:26:46.081214 16104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Bootstrap complete.
I20260928 06:26:46.085073 16104 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:26:46.085430 16104 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc241aa319e41c99aa4c5b2df91fd4e, State: Initialized, Role: FOLLOWER
I20260928 06:26:46.085599 16104 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:26:46.085667 16104 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:26:46.085713 16104 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:26:46.085762 16104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:26:46.087301 16104 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:26:46.087430 16104 leader_election.cc:302] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc241aa319e41c99aa4c5b2df91fd4e; no voters:
I20260928 06:26:46.087600 16104 leader_election.cc:288] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:26:46.087682 16108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:26:46.087924 16108 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 2 LEADER]: Becoming Leader. State: Replica: fdc241aa319e41c99aa4c5b2df91fd4e, State: Running, Role: LEADER
I20260928 06:26:46.087970 16104 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:26:46.088100 16108 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:26:46.088557 16109 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } } }
I20260928 06:26:46.088666 16109 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:46.089350 16110 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdc241aa319e41c99aa4c5b2df91fd4e. Latest consensus state: current_term: 2 leader_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } } }
I20260928 06:26:46.089345 16118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:26:46.089745 16110 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: This master's current role is: LEADER
I20260928 06:26:46.090654 16118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:26:46.090770 16118 catalog_manager.cc:1259] Loaded cluster ID: 5f325166f3784fa0b39b9dd8b86d3bb5
I20260928 06:26:46.090838 16118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:26:46.092072 16118 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:26:46.092267 16118 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Loaded TSK: 0
I20260928 06:26:46.113121 16118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Generated new TSK 1
I20260928 06:26:46.113263 16118 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:26:46.168993 16106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:46.169219 16106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:46.169245 16106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:46.170557 16106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:46.170611 16106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:46.170645 16106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:26:46.172238 16106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:32935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=36975
--tserver_master_addrs=127.31.214.126:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:46.172511 16106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:46.172773 16106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:46.178063 16133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:46.178234 16132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:46.178280 16106 server_base.cc:1034] running on GCE node
W20260928 06:26:46.178400 16135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:46.178632 16106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:46.178966 16106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:46.180166 16106 hybrid_clock.cc:648] HybridClock initialized: now 1790576806180153 us; error 63 us; skew 500 ppm
Sep 28 06:26:46 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:46.183367 16106 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:26:46.183935 16106 webserver.cc:460] Webserver started at http://127.31.214.65:36975/ using document root <none> and password file <none>
I20260928 06:26:46.184216 16106 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:46.184309 16106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:46.185822 16106 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:26:46.186611 16142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:46.186857 16106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:26:46.186982 16106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "60f5f2132f3c49bda8af16c8f9c6e7db"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "4b69b1742b333137bfeef26103dbe9ed"
server_key_iv: "20295fb2295c04a2638ff1c205e728bd"
server_key_version: "encryptionkey@0"
I20260928 06:26:46.187357 16106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:46.197352 16106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:46.200131 16106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:46.200316 16106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:46.200598 16106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:46.201141 16149 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:26:46.203447 16106 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:26:46.203521 16106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 06:26:46.203624 16106 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:26:46.204309 16106 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:26:46.204378 16106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:26:46.204455 16149 tablet_bootstrap.cc:492] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Bootstrap starting.
I20260928 06:26:46.212141 16106 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:32935
I20260928 06:26:46.212195 16256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:32935 every 8 connection(s)
I20260928 06:26:46.212653 16106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:26:46.216912 16149 log.cc:824] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:46.218659 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 16106
I20260928 06:26:46.219002 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:33637
--local_ip_for_outbound_sockets=127.31.214.66
--tserver_master_addrs=127.31.214.126:45905
--webserver_port=36995
--webserver_interface=127.31.214.66
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:46.220844 16149 tablet_bootstrap.cc:492] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:26:46.221233 16149 tablet_bootstrap.cc:492] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Bootstrap complete.
Sep 28 06:26:46 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:46.222711 16149 ts_tablet_manager.cc:1397] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.007s
I20260928 06:26:46.224193 16149 raft_consensus.cc:348] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:46.224534 16149 raft_consensus.cc:729] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f5f2132f3c49bda8af16c8f9c6e7db, State: Initialized, Role: FOLLOWER
I20260928 06:26:46.224674 16149 consensus_queue.cc:260] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } }
I20260928 06:26:46.224980 16149 ts_tablet_manager.cc:1428] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 06:26:46.226939 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.213533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:47275 (local address 127.31.214.126:45905)
0928 06:26:46.214108 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.214118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.214166 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:26:46.214708 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.214713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.214759 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.214899 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:46.215594 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.216549 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.219604 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.220299 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.220422 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.223028 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:46.223033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:46.223033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:46.223055 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:46.225047 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.225503 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.225506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.225507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.225613 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.225878 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.225879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.225879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.226194 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:46.226286 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.226600 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.226725 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":94,"threads_started":1}
I20260928 06:26:46.226943 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.213445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:47275)
0928 06:26:46.213725 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.213771 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.214447 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.214926 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.214931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.215259 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:46.215458 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.215463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.216691 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.216692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.219492 (+ 2800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.219496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.220570 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.220571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.220728 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.221355 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:46.221365 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:46.222911 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:46.225167 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:46.225180 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:46.225182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:46.225404 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:46.225690 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:46.225691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:46.225692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:46.225801 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:46.226280 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:46.226286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:46.226424 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.226534 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.226724 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":114,"threads_started":1}
I20260928 06:26:46.227947 16257 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45905
I20260928 06:26:46.228071 16257 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:46.228256 16257 heartbeater.cc:511] Master 127.31.214.126:45905 requested a full tablet report, sending...
I20260928 06:26:46.229012 16066 ts_manager.cc:194] Registered new tserver with Master: 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65:32935)
I20260928 06:26:46.230248 16066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:47275
I20260928 06:26:46.231148 16257 heartbeater.cc:503] Master 127.31.214.126:45905 was elected leader, sending a full tablet report...
Sep 28 06:26:46 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.65@KRBTEST.COM
I20260928 06:26:46.239656 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.229876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.31.214.65:32935)
0928 06:26:46.230282 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.230286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.230295 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:26:46.230327 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.230329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.230343 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.230403 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:46.231773 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.232539 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.233718 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.234187 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.234274 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.236398 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:46.236407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:46.236407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:46.236419 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:46.238163 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.238683 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.238688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.238689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.238752 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.238966 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.238967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.238967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.239128 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:46.239227 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.239457 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.239562 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":93,"threads_started":1}
I20260928 06:26:46.239656 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.229624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:32935 (local address 127.0.0.1:52218)
0928 06:26:46.230060 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.230098 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.230159 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.230465 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.230469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.231502 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:46.231676 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.231679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.232639 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.232641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.233616 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.233619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.234284 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.234285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.234375 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.234919 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:46.234924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:46.236300 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:46.238298 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:46.238305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:46.238307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:46.238554 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:46.238830 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:46.238831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:46.238831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:46.238900 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:46.239203 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:46.239210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:46.239267 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.239407 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.239542 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":105,"threads_started":1}
I20260928 06:26:46.240411 16191 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e4c35477dc41ab98c05b04ad698268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:25 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:52218
I20260928 06:26:46.241135 16266 tablet_replica.cc:330] stopping tablet replica
I20260928 06:26:46.241271 16266 raft_consensus.cc:2229] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:26:46.241362 16266 raft_consensus.cc:2258] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:46.241703 16266 ts_tablet_manager.cc:1905] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:46.244234 16266 ts_tablet_manager.cc:1918] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:46.244350 16266 log.cc:1196] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal/wals/07e4c35477dc41ab98c05b04ad698268
I20260928 06:26:46.244654 16266 ts_tablet_manager.cc:1939] T 07e4c35477dc41ab98c05b04ad698268 P 60f5f2132f3c49bda8af16c8f9c6e7db: Deleting consensus metadata
I20260928 06:26:46.245249 16055 catalog_manager.cc:4841] TS 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65:32935): tablet 07e4c35477dc41ab98c05b04ad698268 (table test-table [id=7b9c2c28cfdf41e7a2bb779904b19c15]) successfully deleted
W20260928 06:26:46.319461 16262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:46.319676 16262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:46.319701 16262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:46.320899 16262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:46.320945 16262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:46.320973 16262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:26:46.322474 16262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:33637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=36995
--tserver_master_addrs=127.31.214.126:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:46.322715 16262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:46.322968 16262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:46.328429 16272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:46.328462 16274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:46.328433 16271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:46.328454 16262 server_base.cc:1034] running on GCE node
I20260928 06:26:46.328909 16262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:46.329182 16262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:46.330392 16262 hybrid_clock.cc:648] HybridClock initialized: now 1790576806330374 us; error 41 us; skew 500 ppm
Sep 28 06:26:46 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:46.333292 16262 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:26:46.333742 16262 webserver.cc:460] Webserver started at http://127.31.214.66:36995/ using document root <none> and password file <none>
I20260928 06:26:46.334059 16262 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:46.334129 16262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:46.335688 16262 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:46.336444 16281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:46.336776 16262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:46.336854 16262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "d73ad994ff5b49cf90873cd7c692d81a"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "aa11eef1ed65ffe38920b879c1741ae7"
server_key_iv: "73f1e1741ee6de3fab0b105127e85d7b"
server_key_version: "encryptionkey@0"
I20260928 06:26:46.337285 16262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:46.350564 16262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:46.353094 16262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:46.353245 16262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:46.353523 16262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:46.353866 16262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:46.353902 16262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:46.353926 16262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:46.354019 16262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:46.360759 16262 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:33637
I20260928 06:26:46.360800 16394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:33637 every 8 connection(s)
I20260928 06:26:46.361173 16262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:26:46 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:46.370668 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 16262
I20260928 06:26:46.370996 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:37873
--local_ip_for_outbound_sockets=127.31.214.67
--tserver_master_addrs=127.31.214.126:45905
--webserver_port=34833
--webserver_interface=127.31.214.67
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:26:46.371229 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.362055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:56189 (local address 127.31.214.126:45905)
0928 06:26:46.362131 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.362133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.362311 (+ 178us) server_negotiation.cc:407] Connection header received
0928 06:26:46.362795 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.362796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.362810 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.362875 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:46.363856 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.364402 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.365447 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.365682 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.365755 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.368025 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:46.368026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:46.368026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:46.368034 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:46.369504 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.369915 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.369916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.369917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.370058 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.370288 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.370289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.370289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.370400 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:46.370451 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.370972 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.371118 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:26:46.371424 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.361987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:56189)
0928 06:26:46.362166 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.362210 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.362702 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.362930 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.362934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.363512 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:46.363729 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.363734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.364497 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.364498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.365321 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.365324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.365770 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.365771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.365876 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.366515 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:46.366525 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:46.367906 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:46.369605 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:46.369612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:46.369613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:46.369830 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:46.370123 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:46.370124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:46.370124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:46.370227 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:46.370632 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:46.370640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:46.370820 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.370954 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.371142 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260928 06:26:46.372030 16395 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45905
I20260928 06:26:46.372131 16395 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:46.372350 16395 heartbeater.cc:511] Master 127.31.214.126:45905 requested a full tablet report, sending...
I20260928 06:26:46.372786 16066 ts_manager.cc:194] Registered new tserver with Master: d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637)
I20260928 06:26:46.373184 16066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:56189
W20260928 06:26:46.457382 16398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:26:46.457643 16398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:26:46.457670 16398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:26:46.458935 16398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:26:46.458981 16398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:26:46.459012 16398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:26:46.460644 16398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:37873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=34833
--tserver_master_addrs=127.31.214.126:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:26:46.460882 16398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:26:46.461084 16398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:26:46.466943 16403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:26:46.466943 16406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:46.466998 16398 server_base.cc:1034] running on GCE node
W20260928 06:26:46.466939 16404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:26:46.467422 16398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:26:46.467716 16398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:26:46.468917 16398 hybrid_clock.cc:648] HybridClock initialized: now 1790576806468906 us; error 55 us; skew 500 ppm
Sep 28 06:26:46 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:26:46.471937 16398 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:26:46.472456 16398 webserver.cc:460] Webserver started at http://127.31.214.67:34833/ using document root <none> and password file <none>
I20260928 06:26:46.472721 16398 fs_manager.cc:359] Metadata directory not provided
I20260928 06:26:46.472810 16398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:26:46.474305 16398 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:46.475212 16413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:46.475531 16398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:26:46.475649 16398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "c57e4523cdb7bf5eb69b12601615207a"
server_key_iv: "d93183ca2ba0242aa8a73b23d48370f6"
server_key_version: "encryptionkey@0"
I20260928 06:26:46.476039 16398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:26:46.493556 16398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:26:46.496196 16398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:26:46.496371 16398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:26:46.496620 16398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:26:46.497008 16398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:26:46.497066 16398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:46.497109 16398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:26:46.497148 16398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:26:46.503434 16398 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:37873
I20260928 06:26:46.503489 16526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:37873 every 8 connection(s)
I20260928 06:26:46.503861 16398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:26:46.506878 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 16398
Sep 28 06:26:46 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576806, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:26:46.514995 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.507449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.31.214.126:45905)
0928 06:26:46.507603 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.507605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.507616 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:26:46.507700 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.507701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.507714 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.507783 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:46.508555 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.509044 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.510089 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.510334 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.510405 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.511853 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:46.511855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:46.511855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:46.511862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:46.513607 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.514150 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.514157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.514157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.514210 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.514450 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.514451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.514451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.514550 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:46.514676 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.514802 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.514891 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260928 06:26:46.516292 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.504910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:58793 (local address 127.31.214.126:45905)
0928 06:26:46.504962 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.504964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.505210 (+ 246us) server_negotiation.cc:407] Connection header received
0928 06:26:46.505766 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.505767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.505779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.505841 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:46.507179 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.508066 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.509445 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.509735 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.509794 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.512789 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:46.512790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:46.512791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:46.512798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:46.514644 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.515208 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.515209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.515209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.515264 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.515572 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.515573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.515574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.515682 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:46.515761 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.516109 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.516200 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:26:46.516528 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.504857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:58793)
0928 06:26:46.505044 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.505090 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.505626 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.506039 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.506043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.506719 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:26:46.507046 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.507053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.508171 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.508172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.509337 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.509340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.509829 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.509830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.510013 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.510624 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:26:46.510635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:26:46.512682 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:26:46.514775 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:46.514784 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:46.514787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:46.515097 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:46.515335 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:26:46.515336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:26:46.515336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:26:46.515515 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:26:46.515767 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:26:46.515776 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:26:46.515955 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.516088 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.516267 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260928 06:26:46.517038 16527 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45905
I20260928 06:26:46.517162 16527 heartbeater.cc:464] Registering TS with master...
I20260928 06:26:46.517371 16527 heartbeater.cc:511] Master 127.31.214.126:45905 requested a full tablet report, sending...
I20260928 06:26:46.517830 16066 ts_manager.cc:194] Registered new tserver with Master: 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873)
I20260928 06:26:46.518311 16066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:58793
I20260928 06:26:46.526950 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:26:46.530417 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.527454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.31.214.126:45905)
0928 06:26:46.527517 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.527519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.527631 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:26:46.527725 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.527727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.527789 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.527884 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:26:46.528096 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.528744 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.529603 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.529789 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.529848 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.530160 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:26:46.530220 (+ 60us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 06:26:46.530361 (+ 141us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35384: token expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 06:26:46.530508 16260 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35384: token expired
W20260928 06:26:46.530629 15997 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.214.126:45905)
I20260928 06:26:46.537101 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.530885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.31.214.126:45905)
0928 06:26:46.530922 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.530924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.530937 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:26:46.531015 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.531016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.531023 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.531082 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:26:46.531574 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.532025 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.532781 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.532967 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.533021 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.534463 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:26:46.534464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:26:46.534464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:26:46.534471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:26:46.535981 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.536351 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.536352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.536352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.536408 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:26:46.536655 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:26:46.536655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:26:46.536655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:26:46.536754 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:26:46.536829 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.536938 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.536994 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:26:46.538460 16066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:26:46.540099 16066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:26:46.543155 16191 tablet_service.cc:1467] Processing CreateTablet for tablet b8b886fb4c834080a6a699667d5afbd8 (DEFAULT_TABLE table=test-table [id=760abce8679544eb95b4f0d53326d5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:46.543390 16191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b886fb4c834080a6a699667d5afbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:46.544942 16149 tablet_bootstrap.cc:492] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Bootstrap starting.
I20260928 06:26:46.545433 16149 tablet_bootstrap.cc:654] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:46.546252 16149 tablet_bootstrap.cc:492] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: No bootstrap required, opened a new log
I20260928 06:26:46.546320 16149 ts_tablet_manager.cc:1397] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:26:46.546610 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.542602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33637 (local address 127.0.0.1:34982)
0928 06:26:46.542813 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.542826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.542868 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.543198 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.543199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.543201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:46.543246 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.543247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.544414 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.544415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.545401 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.545404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.546352 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.546354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.546436 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.546462 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.546490 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.546536 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260928 06:26:46.546691 16149 raft_consensus.cc:348] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.546780 16149 raft_consensus.cc:374] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:46.546824 16149 raft_consensus.cc:729] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f5f2132f3c49bda8af16c8f9c6e7db, State: Initialized, Role: FOLLOWER
I20260928 06:26:46.546872 16149 consensus_queue.cc:260] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.546943 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.542760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.31.214.66:33637)
0928 06:26:46.543006 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.543012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.543023 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:26:46.543064 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.543066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.543083 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.543167 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:46.543330 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.544320 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.545484 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.546256 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.546723 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.546798 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.546825 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.546864 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":116,"threads_started":1}
I20260928 06:26:46.547138 16149 ts_tablet_manager.cc:1428] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:26:46.547665 16329 tablet_service.cc:1467] Processing CreateTablet for tablet b8b886fb4c834080a6a699667d5afbd8 (DEFAULT_TABLE table=test-table [id=760abce8679544eb95b4f0d53326d5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:46.548033 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b886fb4c834080a6a699667d5afbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:46.548934 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.542683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:37873 (local address 127.0.0.1:36128)
0928 06:26:46.542816 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.542830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.542888 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.545287 (+ 2399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.545288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.545290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:46.545326 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.545327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.546750 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.546751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.547750 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.547753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.548754 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.548755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.548807 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.548810 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.548856 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.548885 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260928 06:26:46.548990 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.542806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.31.214.67:37873)
0928 06:26:46.545028 (+ 2222us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.545033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.545044 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:26:46.545084 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.545086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.545103 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.545195 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:46.545400 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.546657 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.547841 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.548668 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.548749 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.548805 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.548868 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.548910 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2172,"thread_start_us":95,"threads_started":1}
I20260928 06:26:46.549643 16461 tablet_service.cc:1467] Processing CreateTablet for tablet b8b886fb4c834080a6a699667d5afbd8 (DEFAULT_TABLE table=test-table [id=760abce8679544eb95b4f0d53326d5db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:26:46.550045 16461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8b886fb4c834080a6a699667d5afbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:26:46.551895 16537 tablet_bootstrap.cc:492] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: Bootstrap starting.
I20260928 06:26:46.552744 16537 tablet_bootstrap.cc:654] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:46.553248 16537 log.cc:824] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:46.553529 16538 tablet_bootstrap.cc:492] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Bootstrap starting.
I20260928 06:26:46.554075 16537 tablet_bootstrap.cc:492] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: No bootstrap required, opened a new log
I20260928 06:26:46.554203 16537 ts_tablet_manager.cc:1397] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:26:46.554296 16538 tablet_bootstrap.cc:654] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Neither blocks nor log segments found. Creating new log.
I20260928 06:26:46.554970 16538 log.cc:824] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:26:46.555760 16538 tablet_bootstrap.cc:492] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: No bootstrap required, opened a new log
I20260928 06:26:46.555817 16538 ts_tablet_manager.cc:1397] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:26:46.556138 16537 raft_consensus.cc:348] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.556316 16537 raft_consensus.cc:374] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:46.556362 16537 raft_consensus.cc:729] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73ad994ff5b49cf90873cd7c692d81a, State: Initialized, Role: FOLLOWER
I20260928 06:26:46.556486 16537 consensus_queue.cc:260] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.556777 16537 ts_tablet_manager.cc:1428] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:26:46.556851 16395 heartbeater.cc:503] Master 127.31.214.126:45905 was elected leader, sending a full tablet report...
I20260928 06:26:46.557530 16538 raft_consensus.cc:348] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.557700 16538 raft_consensus.cc:374] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:26:46.557745 16538 raft_consensus.cc:729] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c5ff5c67b74ad9b3bc876ad72c15e8, State: Initialized, Role: FOLLOWER
I20260928 06:26:46.557914 16538 consensus_queue.cc:260] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.558256 16538 ts_tablet_manager.cc:1428] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:26:46.558274 16527 heartbeater.cc:503] Master 127.31.214.126:45905 was elected leader, sending a full tablet report...
W20260928 06:26:46.611944 16396 tablet.cc:2369] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:26:46.674717 16541 raft_consensus.cc:482] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:46.674872 16541 raft_consensus.cc:504] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.675244 16541 leader_election.cc:288] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873), 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65:32935)
I20260928 06:26:46.675881 16542 raft_consensus.cc:482] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:26:46.676002 16542 raft_consensus.cc:504] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.676651 16542 leader_election.cc:288] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637), 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65:32935)
I20260928 06:26:46.679309 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.675582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:37873 (local address 127.31.214.66:40039)
0928 06:26:46.675759 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.675768 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.675809 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.675970 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.675971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.675973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:46.676052 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.676054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.676979 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.676980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.678352 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.678355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.679049 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.679050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.679127 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.679131 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.679175 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.679217 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":9,"thread_start_us":129,"threads_started":1}
I20260928 06:26:46.679523 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.675485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:32935 (local address 127.31.214.66:49743)
0928 06:26:46.675554 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.675573 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.675645 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.675888 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.675889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.675891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:46.675972 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.675974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.676980 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.676980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.678352 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.678355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.679271 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.679272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.679359 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.679362 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.679412 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.679449 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":124}
I20260928 06:26:46.679827 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.675778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:40039 (local address 127.31.214.67:37873)
0928 06:26:46.675826 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.675828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.675836 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:26:46.675865 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.675865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.675876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.675942 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:46.676122 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.676872 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.678423 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.678935 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.679006 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.679015 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.679726 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.679766 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:26:46.680102 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b886fb4c834080a6a699667d5afbd8" candidate_uuid: "d73ad994ff5b49cf90873cd7c692d81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" is_pre_election: true
I20260928 06:26:46.680255 16481 raft_consensus.cc:2454] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73ad994ff5b49cf90873cd7c692d81a in term 0.
I20260928 06:26:46.680500 16282 leader_election.cc:302] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c5ff5c67b74ad9b3bc876ad72c15e8, d73ad994ff5b49cf90873cd7c692d81a; no voters:
I20260928 06:26:46.680553 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.676506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:32935 (local address 127.31.214.67:53779)
0928 06:26:46.676556 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.676567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.676603 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.677163 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.677164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.677165 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:46.677218 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.677219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.678243 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.678243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.679492 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.679496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.680320 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.680322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.680398 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.680401 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.680429 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.680479 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":36}
I20260928 06:26:46.680583 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.675666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:49743 (local address 127.31.214.65:32935)
0928 06:26:46.675722 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.675724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.675735 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:26:46.675769 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.675770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.675802 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.675882 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:46.676088 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.676873 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.678458 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.679079 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.679151 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.679236 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.679679 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.680505 (+ 826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":770,"server-negotiator.queue_time_us":31}
I20260928 06:26:46.680641 16541 raft_consensus.cc:2781] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:26:46.680692 16541 raft_consensus.cc:482] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:26:46.680720 16541 raft_consensus.cc:3037] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:46.680853 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.676871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:53779 (local address 127.31.214.65:32935)
0928 06:26:46.677031 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.677033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.677040 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:26:46.677062 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.677063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.677071 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.677282 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:46.677323 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.678137 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.679591 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.680230 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.680680 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.680689 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.680710 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.680745 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260928 06:26:46.680912 16203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b886fb4c834080a6a699667d5afbd8" candidate_uuid: "d73ad994ff5b49cf90873cd7c692d81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" is_pre_election: true
I20260928 06:26:46.680977 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.676575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:50899 (local address 127.31.214.66:33637)
0928 06:26:46.676996 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.676999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.677008 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:26:46.677031 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.677031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.677041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.677095 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:26:46.677326 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.678108 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.680080 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.680609 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.680674 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.680683 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.680869 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.680910 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":359}
I20260928 06:26:46.681054 16203 raft_consensus.cc:2454] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73ad994ff5b49cf90873cd7c692d81a in term 0.
I20260928 06:26:46.681138 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b886fb4c834080a6a699667d5afbd8" candidate_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" is_pre_election: true
I20260928 06:26:46.681195 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.676506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33637 (local address 127.31.214.67:50899)
0928 06:26:46.676818 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.676828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:26:46.676872 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:26:46.677139 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:26:46.677140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:26:46.677142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:26:46.677218 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.677219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.678210 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.678210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.679993 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:26:46.679996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.680717 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.680718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:26:46.680785 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.680789 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:26:46.681054 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 06:26:46.681107 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":163,"thread_start_us":79,"threads_started":1}
I20260928 06:26:46.681219 16202 raft_consensus.cc:2454] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c5ff5c67b74ad9b3bc876ad72c15e8 in term 0.
I20260928 06:26:46.681420 16417 leader_election.cc:302] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c5ff5c67b74ad9b3bc876ad72c15e8, 60f5f2132f3c49bda8af16c8f9c6e7db; no voters:
I20260928 06:26:46.681489 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b886fb4c834080a6a699667d5afbd8" candidate_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73ad994ff5b49cf90873cd7c692d81a" is_pre_election: true
I20260928 06:26:46.681566 16542 raft_consensus.cc:2781] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:26:46.681615 16542 raft_consensus.cc:482] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:26:46.681635 16542 raft_consensus.cc:3037] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:46.681761 16541 raft_consensus.cc:504] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.681905 16541 leader_election.cc:288] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [CANDIDATE]: Term 1 election: Requested vote from peers 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873), 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65:32935)
I20260928 06:26:46.681915 16349 raft_consensus.cc:2379] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46c5ff5c67b74ad9b3bc876ad72c15e8 in current term 1: Already voted for candidate d73ad994ff5b49cf90873cd7c692d81a in this term.
I20260928 06:26:46.682163 16481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b886fb4c834080a6a699667d5afbd8" candidate_uuid: "d73ad994ff5b49cf90873cd7c692d81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8"
I20260928 06:26:46.682216 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b886fb4c834080a6a699667d5afbd8" candidate_uuid: "d73ad994ff5b49cf90873cd7c692d81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db"
I20260928 06:26:46.682291 16202 raft_consensus.cc:3037] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:26:46.682605 16542 raft_consensus.cc:504] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.682711 16481 raft_consensus.cc:2379] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d73ad994ff5b49cf90873cd7c692d81a in current term 1: Already voted for candidate 46c5ff5c67b74ad9b3bc876ad72c15e8 in this term.
I20260928 06:26:46.682761 16542 leader_election.cc:288] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [CANDIDATE]: Term 1 election: Requested vote from peers d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637), 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65:32935)
I20260928 06:26:46.682931 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b886fb4c834080a6a699667d5afbd8" candidate_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73ad994ff5b49cf90873cd7c692d81a"
I20260928 06:26:46.683022 16349 raft_consensus.cc:2379] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 46c5ff5c67b74ad9b3bc876ad72c15e8 in current term 1: Already voted for candidate d73ad994ff5b49cf90873cd7c692d81a in this term.
I20260928 06:26:46.683133 16203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8b886fb4c834080a6a699667d5afbd8" candidate_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db"
I20260928 06:26:46.683439 16202 raft_consensus.cc:2454] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73ad994ff5b49cf90873cd7c692d81a in term 1.
I20260928 06:26:46.683485 16417 leader_election.cc:302] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46c5ff5c67b74ad9b3bc876ad72c15e8; no voters: 60f5f2132f3c49bda8af16c8f9c6e7db, d73ad994ff5b49cf90873cd7c692d81a
I20260928 06:26:46.683602 16542 raft_consensus.cc:2726] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 06:26:46.683663 16285 leader_election.cc:302] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60f5f2132f3c49bda8af16c8f9c6e7db, d73ad994ff5b49cf90873cd7c692d81a; no voters: 46c5ff5c67b74ad9b3bc876ad72c15e8
I20260928 06:26:46.683787 16541 raft_consensus.cc:2781] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:26:46.684010 16541 raft_consensus.cc:686] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 1 LEADER]: Becoming Leader. State: Replica: d73ad994ff5b49cf90873cd7c692d81a, State: Running, Role: LEADER
I20260928 06:26:46.684120 16541 consensus_queue.cc:237] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:26:46.685026 16066 catalog_manager.cc:5495] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a reported cstate change: term changed from 0 to 1, leader changed from <none> to d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66). New cstate: current_term: 1 leader_uuid: "d73ad994ff5b49cf90873cd7c692d81a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } health_report { overall_health: UNKNOWN } } }
I20260928 06:26:46.693399 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.691103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.31.214.66:33637)
0928 06:26:46.691167 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:26:46.691169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:26:46.691198 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:26:46.691238 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:26:46.691239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:26:46.691255 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:26:46.691307 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:26:46.691417 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.691982 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.692689 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:26:46.692871 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:26:46.692925 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:26:46.693065 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:26:46.693168 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:26:46.693229 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:26:46.693262 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:26:46.693278 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:26:46.695247 16481 raft_consensus.cc:1261] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Refusing update from remote peer d73ad994ff5b49cf90873cd7c692d81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:46.695247 16202 raft_consensus.cc:1261] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Refusing update from remote peer d73ad994ff5b49cf90873cd7c692d81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:26:46.695537 16541 consensus_queue.cc:1035] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:46.695709 16546 consensus_queue.cc:1035] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:26:46.698318 16552 mvcc.cc:205] Tried to move back new op lower bound from 7334202600220909568 to 7334202600178925568. Current Snapshot: MvccSnapshot[applied={T|T < 7334202600220909568}]
I20260928 06:26:46.700368 16068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:26:46.700459 16068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:26:46.701297 16068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Sending DeleteTablet for 3 replicas of tablet b8b886fb4c834080a6a699667d5afbd8
I20260928 06:26:46.701660 16461 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b886fb4c834080a6a699667d5afbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:46 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:36128
I20260928 06:26:46.701679 16191 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b886fb4c834080a6a699667d5afbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:46 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:52218
I20260928 06:26:46.701738 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 16106
I20260928 06:26:46.701824 16329 tablet_service.cc:1514] Processing DeleteTablet for tablet b8b886fb4c834080a6a699667d5afbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:26:46 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:34982
I20260928 06:26:46.702114 16559 tablet_replica.cc:330] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: stopping tablet replica
I20260928 06:26:46.702356 16559 raft_consensus.cc:2229] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:26:46.702526 16559 raft_consensus.cc:2258] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:46.702522 16558 tablet_replica.cc:330] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: stopping tablet replica
I20260928 06:26:46.702705 16558 raft_consensus.cc:2229] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:26:46.702916 16558 raft_consensus.cc:2258] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:26:46.703009 16559 ts_tablet_manager.cc:1905] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:46.703454 16558 ts_tablet_manager.cc:1905] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:26:46.705204 16559 ts_tablet_manager.cc:1918] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:46.705288 16559 log.cc:1196] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/b8b886fb4c834080a6a699667d5afbd8
I20260928 06:26:46.705633 16559 ts_tablet_manager.cc:1939] T b8b886fb4c834080a6a699667d5afbd8 P d73ad994ff5b49cf90873cd7c692d81a: Deleting consensus metadata
I20260928 06:26:46.705658 16558 ts_tablet_manager.cc:1918] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:26:46.705881 16558 log.cc:1196] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/b8b886fb4c834080a6a699667d5afbd8
I20260928 06:26:46.706091 16053 catalog_manager.cc:4841] TS d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637): tablet b8b886fb4c834080a6a699667d5afbd8 (table test-table [id=760abce8679544eb95b4f0d53326d5db]) successfully deleted
I20260928 06:26:46.706338 16558 ts_tablet_manager.cc:1939] T b8b886fb4c834080a6a699667d5afbd8 P 46c5ff5c67b74ad9b3bc876ad72c15e8: Deleting consensus metadata
I20260928 06:26:46.706813 16052 catalog_manager.cc:4841] TS 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873): tablet b8b886fb4c834080a6a699667d5afbd8 (table test-table [id=760abce8679544eb95b4f0d53326d5db]) successfully deleted
W20260928 06:26:46.709151 16055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:32935 (error 108)
I20260928 06:26:46.709473 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 16262
I20260928 06:26:46.711670 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:26:46.711494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:32935 (local address 127.0.0.1:52222)
0928 06:26:46.711574 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:26:46.711625 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:32935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 06:26:46.711818 16055 catalog_manager.cc:4584] TS 60f5f2132f3c49bda8af16c8f9c6e7db (127.31.214.65:32935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8b886fb4c834080a6a699667d5afbd8: Network error: Client connection negotiation failed: client connection to 127.31.214.65:32935: connect: Connection refused (error 111)
I20260928 06:26:46.715308 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 16398
I20260928 06:26:46.720467 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 16035
I20260928 06:27:06.726188 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45905
--webserver_interface=127.31.214.126
--webserver_port=34895
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:45905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:06.812678 16561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:06.812927 16561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:06.812971 16561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:06.814210 16561 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:27:06.814270 16561 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:27:06.814292 16561 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:27:06.814311 16561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:06.814339 16561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:06.814371 16561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:06.814400 16561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:06.814430 16561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:06.815989 16561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:45905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:06.816295 16561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:06.816537 16561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:06.822218 16567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:06.822314 16569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:06.822206 16566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:06.822667 16561 server_base.cc:1034] running on GCE node
I20260928 06:27:06.822908 16561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:06.823199 16561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:06.824366 16561 hybrid_clock.cc:648] HybridClock initialized: now 1790576826824354 us; error 32 us; skew 500 ppm
Sep 28 06:27:06 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:06.827106 16561 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:06.827612 16561 webserver.cc:460] Webserver started at http://127.31.214.126:34895/ using document root <none> and password file <none>
I20260928 06:27:06.827840 16561 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:06.827888 16561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:06.829340 16561 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:06.830112 16576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:06.830292 16561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:06.830394 16561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "fdc241aa319e41c99aa4c5b2df91fd4e"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "56eb71b49f28e6995a968bfd3ef9dde5"
server_key_iv: "5fc23339cedbb506f8aab6834a14b1ec"
server_key_version: "encryptionkey@0"
I20260928 06:27:06.830771 16561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:06.844980 16561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:06.847414 16561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:06.847606 16561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:06.852366 16561 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:45905
I20260928 06:27:06.852428 16628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:45905 every 8 connection(s)
I20260928 06:27:06.852880 16561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:06.854813 16629 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:27:06.855593 16629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Bootstrap starting.
I20260928 06:27:06.862191 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 16561
I20260928 06:27:06.862525 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:32935
--local_ip_for_outbound_sockets=127.31.214.65
--tserver_master_addrs=127.31.214.126:45905
--webserver_port=36975
--webserver_interface=127.31.214.65
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:27:06.865491 16629 log.cc:824] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:06.868050 16629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:27:06.868412 16629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Bootstrap complete.
I20260928 06:27:06.870791 16629 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:27:06.871047 16629 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc241aa319e41c99aa4c5b2df91fd4e, State: Initialized, Role: FOLLOWER
I20260928 06:27:06.871241 16629 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:27:06.871333 16629 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:06.871399 16629 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:06.871447 16629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:27:06.872445 16629 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:27:06.872565 16629 leader_election.cc:302] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc241aa319e41c99aa4c5b2df91fd4e; no voters:
I20260928 06:27:06.872764 16629 leader_election.cc:288] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:27:06.872802 16633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:27:06.872944 16633 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [term 3 LEADER]: Becoming Leader. State: Replica: fdc241aa319e41c99aa4c5b2df91fd4e, State: Running, Role: LEADER
I20260928 06:27:06.873065 16629 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:06.873044 16633 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } }
I20260928 06:27:06.873346 16633 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdc241aa319e41c99aa4c5b2df91fd4e. Latest consensus state: current_term: 3 leader_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } } }
I20260928 06:27:06.873420 16633 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:06.873713 16638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:06.873663 16634 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc241aa319e41c99aa4c5b2df91fd4e" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45905 } } }
I20260928 06:27:06.873908 16634 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:06.874637 16638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:06.874749 16638 catalog_manager.cc:1259] Loaded cluster ID: 5f325166f3784fa0b39b9dd8b86d3bb5
I20260928 06:27:06.874785 16638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:06.875972 16638 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:06.876161 16638 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Loaded TSK: 0
I20260928 06:27:06.876212 16638 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fdc241aa319e41c99aa4c5b2df91fd4e: Loaded TSK: 1
F20260928 06:27:06.876454 16638 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa0b74bf9ad google::LogMessage::Fail() at ??:0
@ 0x7fa0b74c288a google::LogMessage::SendToLog() at ??:0
@ 0x7fa0b74bf4de google::LogMessage::Flush() at ??:0
@ 0x7fa0b74c0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa0b9c2c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa0b9c29d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa0bf4191cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa0bf41948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa0bf419594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa0bf3dfc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa0bf3e9ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa0b938d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa0b9382954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa0ba1e66db start_thread at ??:0
@ 0x7fa0b5fbd61f clone at ??:0
W20260928 06:27:06.969903 16631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:06.970212 16631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:06.970254 16631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:06.971472 16631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:06.971541 16631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:06.971587 16631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:27:06.973134 16631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:32935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=36975
--tserver_master_addrs=127.31.214.126:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:06.973435 16631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:06.973681 16631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:06.979028 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:06.979091 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:06.979018 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:06.979586 16631 server_base.cc:1034] running on GCE node
I20260928 06:27:06.979840 16631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:06.980135 16631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:06.981298 16631 hybrid_clock.cc:648] HybridClock initialized: now 1790576826981271 us; error 40 us; skew 500 ppm
Sep 28 06:27:06 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:06.984514 16631 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:27:06.985109 16631 webserver.cc:460] Webserver started at http://127.31.214.65:36975/ using document root <none> and password file <none>
I20260928 06:27:06.985411 16631 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:06.985493 16631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:06.987514 16631 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:06.988445 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:06.988685 16631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:06.988790 16631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "60f5f2132f3c49bda8af16c8f9c6e7db"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "4b69b1742b333137bfeef26103dbe9ed"
server_key_iv: "20295fb2295c04a2638ff1c205e728bd"
server_key_version: "encryptionkey@0"
I20260928 06:27:06.989282 16631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:07.005642 16631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:07.008826 16631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:07.009023 16631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:07.009362 16631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:07.010004 16687 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:27:07.011749 16631 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:27:07.011821 16631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:27:07.011863 16631 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:27:07.012894 16631 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:27:07.012948 16631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:27:07.013070 16687 tablet_bootstrap.cc:492] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Bootstrap starting.
I20260928 06:27:07.021293 16631 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:32935
I20260928 06:27:07.021391 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:32935 every 8 connection(s)
I20260928 06:27:07.021891 16631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:27:07.025039 16687 log.cc:824] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:07.027366 16687 tablet_bootstrap.cc:492] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:27:07.027720 16687 tablet_bootstrap.cc:492] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Bootstrap complete.
I20260928 06:27:07.028700 16687 ts_tablet_manager.cc:1397] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260928 06:27:07.029142 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 16631
I20260928 06:27:07.029512 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:33637
--local_ip_for_outbound_sockets=127.31.214.66
--tserver_master_addrs=127.31.214.126:45905
--webserver_port=36995
--webserver_interface=127.31.214.66
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:27:07.030534 16687 raft_consensus.cc:348] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:27:07.030951 16687 raft_consensus.cc:729] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f5f2132f3c49bda8af16c8f9c6e7db, State: Initialized, Role: FOLLOWER
I20260928 06:27:07.031111 16687 consensus_queue.cc:260] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:27:07.031404 16687 ts_tablet_manager.cc:1428] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260928 06:27:07.099608 16797 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:07.022696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:43707)
0928 06:27:07.022881 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:07.022929 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:07.023733 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:07.099393 (+ 75660us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
W20260928 06:27:07.099934 16795 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:45905 (0 consecutive failures): Network error: Failed to ping master at 127.31.214.126:45905: Client connection negotiation failed: client connection to 127.31.214.126:45905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:27:07.100379 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:07.100200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:46423)
0928 06:27:07.100273 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:07.100341 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:27:07.100741 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:07.100635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:58801)
0928 06:27:07.100685 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:07.100717 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:27:07.100878 16795 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:27:07.116237 16799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:07.116456 16799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:07.116483 16799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:07.117686 16799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:07.117730 16799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:07.117758 16799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:27:07.119573 16799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:33637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=36995
--tserver_master_addrs=127.31.214.126:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:07.119875 16799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:07.120146 16799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:07.125625 16805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:07.125654 16808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:07.125654 16806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:07.125847 16799 server_base.cc:1034] running on GCE node
I20260928 06:27:07.126135 16799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:07.126349 16799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:07.127494 16799 hybrid_clock.cc:648] HybridClock initialized: now 1790576827127475 us; error 35 us; skew 500 ppm
Sep 28 06:27:07 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576827, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:07.130179 16799 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:27:07.130648 16799 webserver.cc:460] Webserver started at http://127.31.214.66:36995/ using document root <none> and password file <none>
I20260928 06:27:07.130908 16799 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:07.130955 16799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:07.132444 16799 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:07.133124 16815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:07.133386 16799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:07.133491 16799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "d73ad994ff5b49cf90873cd7c692d81a"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "aa11eef1ed65ffe38920b879c1741ae7"
server_key_iv: "73f1e1741ee6de3fab0b105127e85d7b"
server_key_version: "encryptionkey@0"
I20260928 06:27:07.133864 16799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:07.150841 16799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:07.153283 16799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:07.153425 16799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:07.153752 16799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:07.154196 16799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:07.154253 16799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:07.154294 16799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:07.154325 16799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:07.160676 16799 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:33637
I20260928 06:27:07.160750 16928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:33637 every 8 connection(s)
I20260928 06:27:07.162457 16799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:27:07.163758 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:07.163306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:37561)
0928 06:27:07.163497 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:07.163573 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
W20260928 06:27:07.164177 16929 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:45905 (0 consecutive failures): Network error: Failed to ping master at 127.31.214.126:45905: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
I20260928 06:27:07.164507 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:07.164386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:41687)
0928 06:27:07.164450 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:07.164484 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:27:07.164814 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:07.164743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:44767)
0928 06:27:07.164768 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:07.164790 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 06:27:07.164927 16929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:27:07.165709 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 16799
I20260928 06:27:07.166049 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:37873
--local_ip_for_outbound_sockets=127.31.214.67
--tserver_master_addrs=127.31.214.126:45905
--webserver_port=34833
--webserver_interface=127.31.214.67
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:07.258606 16932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:07.258896 16932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:07.258947 16932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:07.260252 16932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:07.260337 16932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:07.260387 16932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:27:07.262076 16932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:37873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=34833
--tserver_master_addrs=127.31.214.126:45905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:07.262382 16932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:07.262631 16932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:07.267944 16937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:07.267962 16940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:07.268039 16938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:07.268254 16932 server_base.cc:1034] running on GCE node
I20260928 06:27:07.268535 16932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:07.268788 16932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:07.269932 16932 hybrid_clock.cc:648] HybridClock initialized: now 1790576827269916 us; error 38 us; skew 500 ppm
Sep 28 06:27:07 dist-test-slave-q03f krb5kdc[15487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576827, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:07.272691 16932 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:27:07.273190 16932 webserver.cc:460] Webserver started at http://127.31.214.67:34833/ using document root <none> and password file <none>
I20260928 06:27:07.273432 16932 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:07.273500 16932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:07.274956 16932 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:07.275570 16947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:07.275744 16932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:27:07.275837 16932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8"
format_stamp: "Formatted at 2026-09-28 06:26:25 on dist-test-slave-q03f"
server_key: "c57e4523cdb7bf5eb69b12601615207a"
server_key_iv: "d93183ca2ba0242aa8a73b23d48370f6"
server_key_version: "encryptionkey@0"
I20260928 06:27:07.276193 16932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:07.288168 16932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:07.290584 16932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:07.290753 16932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:07.291008 16932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:07.291386 16932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:07.291443 16932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:07.291488 16932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:07.291515 16932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:07.297842 16932 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:37873
I20260928 06:27:07.297894 17060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:37873 every 8 connection(s)
I20260928 06:27:07.298295 16932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:27:07.299475 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:07.298974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:42053)
0928 06:27:07.299159 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:07.299318 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260928 06:27:07.299942 17061 heartbeater.cc:644] Failed to heartbeat to 127.31.214.126:45905 (0 consecutive failures): Network error: Failed to ping master at 127.31.214.126:45905: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
I20260928 06:27:07.300304 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:07.300149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:50709)
0928 06:27:07.300194 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:07.300223 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:27:07.300549 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:07.300499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:42379)
0928 06:27:07.300519 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:07.300537 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 06:27:07.300661 17061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:27:07.301510 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 16932
I20260928 06:27:08.101727 17065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:08.101452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:53661)
0928 06:27:08.101606 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:08.101658 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260928 06:27:08.165642 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:08.165367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:42995)
0928 06:27:08.165526 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:08.165575 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260928 06:27:08.301367 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:08.301068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:44793)
0928 06:27:08.301202 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:08.301307 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260928 06:27:08.801398 17068 raft_consensus.cc:482] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:08.801548 17068 raft_consensus.cc:504] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:27:08.802009 17068 leader_election.cc:288] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637), 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873)
Sep 28 06:27:08 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.66@KRBTEST.COM
Sep 28 06:27:08 dist-test-slave-q03f krb5kdc[15487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576826, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.67@KRBTEST.COM
I20260928 06:27:08.813217 17071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:08.802330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:33587 (local address 127.31.214.66:33637)
0928 06:27:08.802493 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:08.802507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:08.802533 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:27:08.803029 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:08.803032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:08.803074 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:08.803235 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:08.804018 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:08.804888 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:08.805937 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:08.806405 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:08.806536 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:08.809228 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:08.809233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:08.809233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:08.809246 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:08.811237 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:08.811828 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:08.811832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:08.811832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:08.811960 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:08.812250 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:08.812251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:08.812252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:08.812536 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:08.812599 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:08.812968 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 06:27:08.813098 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":55,"threads_started":1}
I20260928 06:27:08.813205 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:08.802209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:37873 (local address 127.31.214.65:46037)
0928 06:27:08.802437 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:08.802451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:08.802514 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:08.803306 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:08.803307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:08.803697 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:08.803880 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:08.803886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:08.805013 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:08.805014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:08.805797 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:08.805799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:08.806469 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:08.806470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:08.806581 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:08.807258 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:08.807263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:08.809391 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:08.811597 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:08.811597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:08.811598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:08.811880 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:08.812177 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:08.812178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:08.812178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:08.812266 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:08.812692 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:08.812693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:08.812772 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 06:27:08.812861 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:27:08.813087 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":56,"thread_start_us":88,"threads_started":1}
I20260928 06:27:08.813241 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:08.802401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:46037 (local address 127.31.214.67:37873)
0928 06:27:08.802578 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:08.802588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:08.802609 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:27:08.803028 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:08.803031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:08.803074 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:08.803321 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:08.804012 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:08.804892 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:08.805912 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:08.806379 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:08.806461 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:08.809523 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:08.809527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:08.809528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:08.809539 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:08.811504 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:08.811965 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:08.811967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:08.811967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:08.812103 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:08.812337 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:08.812338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:08.812338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:08.812603 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:08.812694 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:08.812952 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 06:27:08.813112 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260928 06:27:08.813405 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:08.802158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33637 (local address 127.31.214.65:33587)
0928 06:27:08.802382 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:08.802394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:08.802458 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:08.803281 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:08.803284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:08.803696 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:08.803881 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:08.803886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:08.804996 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:08.804997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:08.805796 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:08.805799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:08.806514 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:08.806515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:08.806592 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:08.807221 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:08.807231 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:08.809093 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:08.811335 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:08.811344 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:08.811347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:08.811718 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:08.812030 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:08.812031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:08.812032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:08.812180 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:08.812610 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:08.812618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:08.812799 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 06:27:08.812893 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:27:08.813171 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":147,"thread_start_us":94,"threads_started":1}
W20260928 06:27:08.814834 16681 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
W20260928 06:27:08.814952 16682 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
I20260928 06:27:08.815002 16682 leader_election.cc:302] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60f5f2132f3c49bda8af16c8f9c6e7db; no voters: 46c5ff5c67b74ad9b3bc876ad72c15e8, d73ad994ff5b49cf90873cd7c692d81a
I20260928 06:27:08.815130 17068 raft_consensus.cc:2726] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:27:09.102563 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:09.102391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:58979)
0928 06:27:09.102468 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:09.102520 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 06:27:09.166529 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:09.166223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:40497)
0928 06:27:09.166395 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:09.166471 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260928 06:27:09.302400 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:09.302082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:52633)
0928 06:27:09.302266 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:09.302323 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260928 06:27:10.103575 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:10.103297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:35667)
0928 06:27:10.103446 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:10.103495 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260928 06:27:10.167402 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:10.167095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:47125)
0928 06:27:10.167256 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:10.167349 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260928 06:27:10.303488 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:10.303248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:39425)
0928 06:27:10.303390 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:10.303436 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260928 06:27:10.691383 17078 raft_consensus.cc:482] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:10.691486 17078 raft_consensus.cc:504] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:27:10.691715 17078 leader_election.cc:288] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637), 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873)
W20260928 06:27:10.692315 16682 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
W20260928 06:27:10.692442 16681 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
I20260928 06:27:10.692533 16681 leader_election.cc:302] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60f5f2132f3c49bda8af16c8f9c6e7db; no voters: 46c5ff5c67b74ad9b3bc876ad72c15e8, d73ad994ff5b49cf90873cd7c692d81a
I20260928 06:27:10.692618 17078 raft_consensus.cc:2726] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:27:11.104575 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:11.104288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:44831)
0928 06:27:11.104472 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:11.104523 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260928 06:27:11.168344 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:11.168033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:46425)
0928 06:27:11.168224 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:11.168276 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260928 06:27:11.304333 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:11.304067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:46325)
0928 06:27:11.304229 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:11.304280 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260928 06:27:12.105417 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:12.105154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:36319)
0928 06:27:12.105312 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:12.105363 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260928 06:27:12.169272 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:12.168948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:48843)
0928 06:27:12.169171 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:12.169221 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":67,"threads_started":1}
I20260928 06:27:12.305197 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:12.304890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:57427)
0928 06:27:12.305086 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:12.305136 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260928 06:27:12.762630 17085 raft_consensus.cc:482] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:12.762753 17085 raft_consensus.cc:504] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:27:12.762997 17085 leader_election.cc:288] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637), 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873)
W20260928 06:27:12.763459 16681 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
W20260928 06:27:12.763552 16682 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
I20260928 06:27:12.763571 16682 leader_election.cc:302] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60f5f2132f3c49bda8af16c8f9c6e7db; no voters: 46c5ff5c67b74ad9b3bc876ad72c15e8, d73ad994ff5b49cf90873cd7c692d81a
I20260928 06:27:12.763648 17085 raft_consensus.cc:2726] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:27:13.106235 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:13.105933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:50695)
0928 06:27:13.106137 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:13.106183 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260928 06:27:13.170136 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:13.169811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:50577)
0928 06:27:13.170043 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:13.170087 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":65,"threads_started":1}
I20260928 06:27:13.306154 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:13.305833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:58331)
0928 06:27:13.306055 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:13.306099 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":90,"threads_started":1}
I20260928 06:27:14.107079 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:14.106825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:47387)
0928 06:27:14.106969 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:14.107021 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260928 06:27:14.171116 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:14.170856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:58803)
0928 06:27:14.171007 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:14.171053 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260928 06:27:14.306995 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:14.306702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:41787)
0928 06:27:14.306865 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:14.306926 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260928 06:27:15.107851 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:15.107621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:58077)
0928 06:27:15.107761 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:15.107800 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260928 06:27:15.171875 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:15.171650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:33271)
0928 06:27:15.171785 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:15.171826 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260928 06:27:15.307857 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:15.307571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:38441)
0928 06:27:15.307760 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:15.307809 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260928 06:27:15.730798 17095 raft_consensus.cc:482] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:15.730913 17095 raft_consensus.cc:504] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:27:15.731124 17095 leader_election.cc:288] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637), 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873)
W20260928 06:27:15.731530 16682 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
W20260928 06:27:15.731599 16681 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
I20260928 06:27:15.731685 16681 leader_election.cc:302] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60f5f2132f3c49bda8af16c8f9c6e7db; no voters: 46c5ff5c67b74ad9b3bc876ad72c15e8, d73ad994ff5b49cf90873cd7c692d81a
I20260928 06:27:15.731766 17095 raft_consensus.cc:2726] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:27:16.108739 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:16.108491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:41151)
0928 06:27:16.108638 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:16.108689 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260928 06:27:16.172724 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:16.172418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:44431)
0928 06:27:16.172590 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:16.172641 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260928 06:27:16.308785 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:16.308511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:40073)
0928 06:27:16.308685 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:16.308736 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260928 06:27:17.109747 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:17.109473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:44981)
0928 06:27:17.109650 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:17.109696 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":74,"threads_started":1}
I20260928 06:27:17.173622 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:17.173347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:38445)
0928 06:27:17.173530 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:17.173571 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20260928 06:27:17.309654 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:17.309369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:43391)
0928 06:27:17.309547 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:17.309598 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260928 06:27:18.110613 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:18.110380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:36395)
0928 06:27:18.110513 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:18.110561 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260928 06:27:18.174479 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:18.174234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:45109)
0928 06:27:18.174380 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:18.174427 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260928 06:27:18.310578 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:18.310303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:57343)
0928 06:27:18.310456 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:18.310500 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260928 06:27:18.978879 17105 raft_consensus.cc:482] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:18.978986 17105 raft_consensus.cc:504] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ad994ff5b49cf90873cd7c692d81a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33637 } } peers { permanent_uuid: "46c5ff5c67b74ad9b3bc876ad72c15e8" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37873 } } peers { permanent_uuid: "60f5f2132f3c49bda8af16c8f9c6e7db" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 32935 } }
I20260928 06:27:18.979153 17105 leader_election.cc:288] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637), 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873)
W20260928 06:27:18.979621 16681 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46c5ff5c67b74ad9b3bc876ad72c15e8 (127.31.214.67:37873): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
W20260928 06:27:18.979727 16682 leader_election.cc:341] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d73ad994ff5b49cf90873cd7c692d81a (127.31.214.66:33637): Not found: Tablet not found: b8b886fb4c834080a6a699667d5afbd8
I20260928 06:27:18.979763 16682 leader_election.cc:302] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60f5f2132f3c49bda8af16c8f9c6e7db; no voters: 46c5ff5c67b74ad9b3bc876ad72c15e8, d73ad994ff5b49cf90873cd7c692d81a
I20260928 06:27:18.979869 17105 raft_consensus.cc:2726] T b8b886fb4c834080a6a699667d5afbd8 P 60f5f2132f3c49bda8af16c8f9c6e7db [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:27:19.111745 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:19.111435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:48327)
0928 06:27:19.111644 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:19.111695 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":104,"threads_started":1}
I20260928 06:27:19.175453 17107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:19.175111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:51781)
0928 06:27:19.175348 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:19.175400 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":126,"threads_started":1}
I20260928 06:27:19.311489 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:19.311209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:55831)
0928 06:27:19.311389 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:19.311441 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260928 06:27:20.112581 17109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:20.112342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:52377)
0928 06:27:20.112484 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:20.112528 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260928 06:27:20.176348 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:20.176079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:45901)
0928 06:27:20.176254 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:20.176295 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260928 06:27:20.312367 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:20.312079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:34535)
0928 06:27:20.312264 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:20.312315 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260928 06:27:21.113440 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:21.113203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:55993)
0928 06:27:21.113343 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:21.113391 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260928 06:27:21.177232 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:21.176984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:60073)
0928 06:27:21.177125 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:21.177180 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260928 06:27:21.313278 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:21.312991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:56493)
0928 06:27:21.313172 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:21.313224 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260928 06:27:22.114594 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:22.114343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.65:42687)
0928 06:27:22.114499 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:22.114547 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260928 06:27:22.178287 17116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:22.178023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.66:32877)
0928 06:27:22.178186 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:22.178236 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260928 06:27:22.311344 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 16631
I20260928 06:27:22.314289 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:22.314073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45905 (local address 127.31.214.67:43515)
0928 06:27:22.314204 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:22.314242 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.126:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260928 06:27:22.316992 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 16799
I20260928 06:27:22.321331 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 16932
2026-09-28T06:27:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58473 ms)
[----------] 30 tests from SecurityITest (272201 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:22 dist-test-slave-q03f krb5kdc[17120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:22 dist-test-slave-q03f krb5kdc[17120](info): set up 2 sockets
Sep 28 06:27:22 dist-test-slave-q03f krb5kdc[17120](info): commencing operation
krb5kdc: starting...
W20260928 06:27:23.385728 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:23 dist-test-slave-q03f krb5kdc[17120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:23.511051 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45663
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:23.594146 17133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:23.594406 17133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:23.594444 17133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:23.595644 17133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:23.595714 17133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:23.595741 17133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:23.595770 17133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:23.595800 17133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:23.597360 17133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:23.597658 17133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:23.597939 17133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:23.603471 17141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:23.603518 17138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:23.603538 17139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:23.603544 17133 server_base.cc:1034] running on GCE node
I20260928 06:27:23.604086 17133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:23.604413 17133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:23.605579 17133 hybrid_clock.cc:648] HybridClock initialized: now 1790576843605562 us; error 46 us; skew 500 ppm
Sep 28 06:27:23 dist-test-slave-q03f krb5kdc[17120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576843, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:23.608532 17133 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:23.609042 17133 webserver.cc:460] Webserver started at http://127.31.214.126:41017/ using document root <none> and password file <none>
I20260928 06:27:23.609308 17133 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:23.609378 17133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:23.609540 17133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:23.610736 17133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "8978b7a239dc432887f486fb48a556e3"
format_stamp: "Formatted at 2026-09-28 06:27:23 on dist-test-slave-q03f"
server_key: "07374273dbe055352068c5dd499ea30d"
server_key_iv: "43eb47410343a28cd087dcb27995f97c"
server_key_version: "encryptionkey@0"
I20260928 06:27:23.611148 17133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "8978b7a239dc432887f486fb48a556e3"
format_stamp: "Formatted at 2026-09-28 06:27:23 on dist-test-slave-q03f"
server_key: "07374273dbe055352068c5dd499ea30d"
server_key_iv: "43eb47410343a28cd087dcb27995f97c"
server_key_version: "encryptionkey@0"
I20260928 06:27:23.612665 17133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:27:23.613473 17148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:23.613680 17133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:23.613792 17133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "8978b7a239dc432887f486fb48a556e3"
format_stamp: "Formatted at 2026-09-28 06:27:23 on dist-test-slave-q03f"
server_key: "07374273dbe055352068c5dd499ea30d"
server_key_iv: "43eb47410343a28cd087dcb27995f97c"
server_key_version: "encryptionkey@0"
I20260928 06:27:23.613870 17133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:23.631461 17133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:23.633531 17133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:23.633744 17133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:23.638062 17133 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:45663
I20260928 06:27:23.638144 17200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:45663 every 8 connection(s)
I20260928 06:27:23.638525 17133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:23.639443 17201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:23.641913 17201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3: Bootstrap starting.
I20260928 06:27:23.642630 17201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:23.642928 17201 log.cc:824] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:23.643743 17201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3: No bootstrap required, opened a new log
I20260928 06:27:23.645455 17201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8978b7a239dc432887f486fb48a556e3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45663 } }
I20260928 06:27:23.645610 17201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:23.645634 17201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8978b7a239dc432887f486fb48a556e3, State: Initialized, Role: FOLLOWER
I20260928 06:27:23.645805 17201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8978b7a239dc432887f486fb48a556e3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45663 } }
I20260928 06:27:23.645895 17201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:23.645931 17201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:23.646011 17201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:23.646277 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 17133
I20260928 06:27:23.646406 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:23.646610 32601 external_mini_cluster.cc:1428] Setting key 2d1d6859f1ca7f1f0a42eff763b48927
I20260928 06:27:23.646685 17201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8978b7a239dc432887f486fb48a556e3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45663 } }
I20260928 06:27:23.646844 17201 leader_election.cc:302] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8978b7a239dc432887f486fb48a556e3; no voters:
I20260928 06:27:23.647044 17201 leader_election.cc:288] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:23.647090 17204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:23.647356 17204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [term 1 LEADER]: Becoming Leader. State: Replica: 8978b7a239dc432887f486fb48a556e3, State: Running, Role: LEADER
I20260928 06:27:23.647418 17201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:23.647503 17204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8978b7a239dc432887f486fb48a556e3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45663 } }
I20260928 06:27:23.647990 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8978b7a239dc432887f486fb48a556e3. Latest consensus state: current_term: 1 leader_uuid: "8978b7a239dc432887f486fb48a556e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8978b7a239dc432887f486fb48a556e3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45663 } } }
I20260928 06:27:23.648103 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:23.648132 17206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8978b7a239dc432887f486fb48a556e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8978b7a239dc432887f486fb48a556e3" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45663 } } }
I20260928 06:27:23.648212 17206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3 [sys.catalog]: This master's current role is: LEADER
W20260928 06:27:23.649403 17222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:27:23.649461 17222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:27:23.649503 17215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:23.649654 17215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:23.651415 17215 catalog_manager.cc:1347] Generated new cluster ID: 834b24201b5c4490aea7691b1c69c620
I20260928 06:27:23.651470 17215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:27:23 dist-test-slave-q03f krb5kdc[17120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:23.659406 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:23.647210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.31.214.126:45663)
0928 06:27:23.647418 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:23.647428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:23.647464 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:27:23.647587 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:23.647595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:23.647657 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:23.648689 (+ 1032us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:23.648802 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:23.650288 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:23.651253 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:23.651879 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:23.651996 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:23.654803 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:23.654807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:23.654807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:23.654820 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:23.657568 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:23.658010 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:23.658013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:23.658014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:23.658148 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:23.658362 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:23.658363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:23.658363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:23.658742 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:23.658927 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:23.658998 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:27:23.659161 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
I20260928 06:27:23.664945 17215 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:23.665619 17215 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:23.673094 17215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3: Generated new TSK 0
I20260928 06:27:23.673336 17215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:27:23.705610 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45663
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:23.792090 17227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:23.792351 17227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:23.792393 17227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:23.793581 17227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:23.793658 17227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:23.793704 17227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:27:23.795265 17227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:23.795542 17227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:23.795797 17227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:23.801334 17233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:23.801366 17232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:23.801321 17235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:23.801859 17227 server_base.cc:1034] running on GCE node
I20260928 06:27:23.802147 17227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:23.802409 17227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:23.803588 17227 hybrid_clock.cc:648] HybridClock initialized: now 1790576843803570 us; error 59 us; skew 500 ppm
Sep 28 06:27:23 dist-test-slave-q03f krb5kdc[17120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576843, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:23.806269 17227 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:27:23.806746 17227 webserver.cc:460] Webserver started at http://127.31.214.65:33595/ using document root <none> and password file <none>
I20260928 06:27:23.806981 17227 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:23.807052 17227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:23.807211 17227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:23.808336 17227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "5c55841b24a14105a4cabb4b70cec721"
format_stamp: "Formatted at 2026-09-28 06:27:23 on dist-test-slave-q03f"
server_key: "77541a28d2384135251b0f79bb088fd8"
server_key_iv: "f5ff46f835e7b4709a59d01ef73100aa"
server_key_version: "encryptionkey@0"
I20260928 06:27:23.808713 17227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "5c55841b24a14105a4cabb4b70cec721"
format_stamp: "Formatted at 2026-09-28 06:27:23 on dist-test-slave-q03f"
server_key: "77541a28d2384135251b0f79bb088fd8"
server_key_iv: "f5ff46f835e7b4709a59d01ef73100aa"
server_key_version: "encryptionkey@0"
I20260928 06:27:23.810307 17227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:27:23.811261 17242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:23.811460 17227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:23.811612 17227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "5c55841b24a14105a4cabb4b70cec721"
format_stamp: "Formatted at 2026-09-28 06:27:23 on dist-test-slave-q03f"
server_key: "77541a28d2384135251b0f79bb088fd8"
server_key_iv: "f5ff46f835e7b4709a59d01ef73100aa"
server_key_version: "encryptionkey@0"
I20260928 06:27:23.811702 17227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:23.822836 17227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:23.824890 17227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:23.825085 17227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:23.825361 17227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:23.825712 17227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:23.825773 17227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:23.825821 17227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:23.825875 17227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:23.832744 17227 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:36161
I20260928 06:27:23.832784 17355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:36161 every 8 connection(s)
I20260928 06:27:23.833189 17227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:27:23 dist-test-slave-q03f krb5kdc[17120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576843, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:23.841548 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 17227
I20260928 06:27:23.841647 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:27:23.841868 32601 external_mini_cluster.cc:1428] Setting key 5d7e3002f8126b1f0f3125539122a5f2
I20260928 06:27:23.844187 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:23.834068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:55745 (local address 127.31.214.126:45663)
0928 06:27:23.834143 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:23.834146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:23.834395 (+ 249us) server_negotiation.cc:407] Connection header received
0928 06:27:23.835018 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:23.835020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:23.835037 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:23.835101 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:23.836198 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:23.836819 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:23.837996 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:23.838248 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:23.838348 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:23.840703 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:23.840704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:23.840705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:23.840714 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:23.842548 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:23.843071 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:23.843073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:23.843074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:23.843149 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:23.843449 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:23.843450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:23.843450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:23.843582 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:23.843667 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:23.843989 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 06:27:23.844085 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:27:23.844450 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:23.834073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45663 (local address 127.31.214.65:55745)
0928 06:27:23.834250 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:23.834291 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:23.834882 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:23.835184 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:23.835188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:23.835851 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:23.836076 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:23.836081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:23.836938 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:23.836940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:23.837862 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:23.837864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:23.838364 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:23.838365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:23.838518 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:23.839090 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:23.839099 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:23.840589 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:23.842623 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:23.842632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:23.842635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:23.842890 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:23.843239 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:23.843239 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:23.843240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:23.843366 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:23.843657 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:23.843665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:23.843829 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 06:27:23.843957 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 06:27:23.844148 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260928 06:27:23.845022 17356 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45663
I20260928 06:27:23.845157 17356 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:23.845368 17356 heartbeater.cc:511] Master 127.31.214.126:45663 requested a full tablet report, sending...
I20260928 06:27:23.846032 17165 ts_manager.cc:194] Registered new tserver with Master: 5c55841b24a14105a4cabb4b70cec721 (127.31.214.65:36161)
I20260928 06:27:23.847046 17165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:55745
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:27:23.869513 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45663
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:23.955252 17361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:23.955497 17361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:23.955524 17361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:23.956791 17361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:23.956838 17361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:23.956866 17361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:27:23.958333 17361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:23.958534 17361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:23.958721 17361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:23.964403 17369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:23.964376 17367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:23.964390 17366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:23.964749 17361 server_base.cc:1034] running on GCE node
I20260928 06:27:23.964973 17361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:23.965250 17361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:23.966419 17361 hybrid_clock.cc:648] HybridClock initialized: now 1790576843966406 us; error 33 us; skew 500 ppm
Sep 28 06:27:23 dist-test-slave-q03f krb5kdc[17120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576843, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:23.969147 17361 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:27:23.969652 17361 webserver.cc:460] Webserver started at http://127.31.214.66:37831/ using document root <none> and password file <none>
I20260928 06:27:23.969885 17361 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:23.969972 17361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:23.970139 17361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:23.971217 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "dd1c50d800714b7c9205b9de5fc3aafb"
format_stamp: "Formatted at 2026-09-28 06:27:23 on dist-test-slave-q03f"
server_key: "7efcde5dca83cb530e4a4e16d1295903"
server_key_iv: "a6242158817989ab9522412aa1aded32"
server_key_version: "encryptionkey@0"
I20260928 06:27:23.971597 17361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "dd1c50d800714b7c9205b9de5fc3aafb"
format_stamp: "Formatted at 2026-09-28 06:27:23 on dist-test-slave-q03f"
server_key: "7efcde5dca83cb530e4a4e16d1295903"
server_key_iv: "a6242158817989ab9522412aa1aded32"
server_key_version: "encryptionkey@0"
I20260928 06:27:23.973093 17361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:27:23.974090 17376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:23.974308 17361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:23.974444 17361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "dd1c50d800714b7c9205b9de5fc3aafb"
format_stamp: "Formatted at 2026-09-28 06:27:23 on dist-test-slave-q03f"
server_key: "7efcde5dca83cb530e4a4e16d1295903"
server_key_iv: "a6242158817989ab9522412aa1aded32"
server_key_version: "encryptionkey@0"
I20260928 06:27:23.974557 17361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:24.010499 17361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:24.013200 17361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:24.013389 17361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:24.013671 17361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:24.014118 17361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:24.014180 17361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.014228 17361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:24.014259 17361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.021396 17361 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:37521
I20260928 06:27:24.021456 17489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:37521 every 8 connection(s)
I20260928 06:27:24.021806 17361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:27:24.025453 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 17361
I20260928 06:27:24.025554 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:27:24.025779 32601 external_mini_cluster.cc:1428] Setting key 54d6f477e0a9e1792460643cfb037329
Sep 28 06:27:24 dist-test-slave-q03f krb5kdc[17120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576843, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:24.032670 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.022623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:60273 (local address 127.31.214.126:45663)
0928 06:27:24.022724 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.022726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.022873 (+ 147us) server_negotiation.cc:407] Connection header received
0928 06:27:24.023389 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.023390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.023401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.023468 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:24.024451 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.024978 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.026242 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.026617 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.026694 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.029584 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:24.029586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:24.029586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:24.029601 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:24.031154 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:24.031610 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:24.031611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:24.031612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:24.031678 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:24.031960 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:24.031961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:24.031962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:24.032087 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:24.032167 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.032466 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.032569 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 06:27:24.032900 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.022585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45663 (local address 127.31.214.66:60273)
0928 06:27:24.022750 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.022795 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:24.023303 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:24.023540 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:24.023544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:24.024144 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:24.024349 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.024355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.025064 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.025065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.026140 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.026144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.026712 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.026713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.026877 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.027473 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:24.027483 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:24.029470 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:24.031255 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:24.031267 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:24.031270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:24.031526 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:24.031756 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:24.031757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:24.031757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:24.031885 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:24.032158 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:24.032166 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:24.032328 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 06:27:24.032444 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:27:24.032634 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260928 06:27:24.033402 17490 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45663
I20260928 06:27:24.033504 17490 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:24.033725 17490 heartbeater.cc:511] Master 127.31.214.126:45663 requested a full tablet report, sending...
I20260928 06:27:24.034158 17165 ts_manager.cc:194] Registered new tserver with Master: dd1c50d800714b7c9205b9de5fc3aafb (127.31.214.66:37521)
I20260928 06:27:24.034744 17165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:60273
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:27:24.059561 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45663
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:24.145249 17495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:24.145485 17495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:24.145514 17495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:24.146878 17495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:24.146931 17495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:24.146970 17495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:27:24.149058 17495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:24.149326 17495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:24.149554 17495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:24.155048 17501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:24.155100 17500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:24.155112 17503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:24.155597 17495 server_base.cc:1034] running on GCE node
I20260928 06:27:24.155797 17495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:24.156051 17495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:24.157191 17495 hybrid_clock.cc:648] HybridClock initialized: now 1790576844157178 us; error 28 us; skew 500 ppm
Sep 28 06:27:24 dist-test-slave-q03f krb5kdc[17120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576844, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:24.159804 17495 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:27:24.160267 17495 webserver.cc:460] Webserver started at http://127.31.214.67:34643/ using document root <none> and password file <none>
I20260928 06:27:24.160521 17495 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:24.160588 17495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:24.160735 17495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:24.161815 17495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "6b406edbf8b04391bc2ed41f1e9a12fb"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "3add699bacca28d8432e953017d383f8"
server_key_iv: "e44a9ddecdf055a5dc3eac3ff6a93ab0"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.162220 17495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "6b406edbf8b04391bc2ed41f1e9a12fb"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "3add699bacca28d8432e953017d383f8"
server_key_iv: "e44a9ddecdf055a5dc3eac3ff6a93ab0"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.163558 17495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:24.164336 17510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.164526 17495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:24.164645 17495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "6b406edbf8b04391bc2ed41f1e9a12fb"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "3add699bacca28d8432e953017d383f8"
server_key_iv: "e44a9ddecdf055a5dc3eac3ff6a93ab0"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.164717 17495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:24.176921 17495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:24.179420 17495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:24.179589 17495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:24.179837 17495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:24.180157 17495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:24.180210 17495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.180255 17495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:24.180281 17495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.186476 17495 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:38993
I20260928 06:27:24.186527 17623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:38993 every 8 connection(s)
I20260928 06:27:24.186911 17495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:27:24 dist-test-slave-q03f krb5kdc[17120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576844, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:24.195488 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 17495
I20260928 06:27:24.195616 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:27:24.195811 32601 external_mini_cluster.cc:1428] Setting key 10f743b186e002f26904bf1a3df9a9d2
I20260928 06:27:24.197114 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.187783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:46749 (local address 127.31.214.126:45663)
0928 06:27:24.187878 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.187880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.188129 (+ 249us) server_negotiation.cc:407] Connection header received
0928 06:27:24.188647 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.188648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.188660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.188719 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:24.189742 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.190318 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.191350 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.191551 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.191633 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.193913 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:24.193915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:24.193915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:24.193922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:24.195365 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:24.195902 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:24.195904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:24.195904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:24.195997 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:24.196365 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:24.196366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:24.196366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:24.196550 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:24.196623 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.196955 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.197028 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:27:24.197333 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.187746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45663 (local address 127.31.214.67:46749)
0928 06:27:24.187932 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.187978 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:24.188524 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:24.188794 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:24.188800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:24.189390 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:24.189627 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.189632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.190420 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.190421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.191251 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.191253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.191640 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.191642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.191778 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.192367 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:24.192376 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:24.193779 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:24.195485 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:24.195493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:24.195496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:24.195820 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:24.196093 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:24.196094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:24.196094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:24.196293 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:24.196628 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:24.196634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:24.196807 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 06:27:24.196927 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 06:27:24.197066 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260928 06:27:24.197868 17624 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45663
I20260928 06:27:24.198007 17624 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:24.198184 17624 heartbeater.cc:511] Master 127.31.214.126:45663 requested a full tablet report, sending...
I20260928 06:27:24.198537 17165 ts_manager.cc:194] Registered new tserver with Master: 6b406edbf8b04391bc2ed41f1e9a12fb (127.31.214.67:38993)
I20260928 06:27:24.198899 17165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:46749
I20260928 06:27:24.207820 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:27:24.215603 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.209218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.31.214.126:45663)
0928 06:27:24.209313 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.209315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.209390 (+ 75us) server_negotiation.cc:407] Connection header received
0928 06:27:24.209451 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.209452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.209461 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.209501 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:24.210259 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.210752 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.211501 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.211689 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.211739 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.213035 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:24.213037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:24.213037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:24.213045 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:24.214488 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:24.214839 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:24.214840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:24.214851 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:24.214911 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:24.215104 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:24.215104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:24.215104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:24.215257 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:24.215334 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.215422 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.215486 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 06:27:24.217514 17165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:27:24.219254 17165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:27:24.226248 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.221809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:36161 (local address 127.0.0.1:40444)
0928 06:27:24.221998 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.222039 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:24.222182 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:24.222516 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:24.222519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:24.222526 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:24.222606 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.222608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.224044 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.224045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.224966 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.224969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.225926 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.225927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.226027 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.226031 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:24.226063 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 06:27:24.226146 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":65,"threads_started":1}
I20260928 06:27:24.226262 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.222031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.31.214.66:37521)
0928 06:27:24.222341 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.222345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.222355 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:27:24.222390 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.222392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.222406 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.222783 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:24.222862 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.223900 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.225302 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.225788 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.225878 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.225945 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.226130 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.226175 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":123,"threads_started":1}
I20260928 06:27:24.226258 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.221821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:37521 (local address 127.0.0.1:55798)
0928 06:27:24.222158 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.222170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:24.222225 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:24.222652 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:24.222653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:24.222655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:24.222703 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.222704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.224008 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.224009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.225208 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.225210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.225877 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.225878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.225996 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.226011 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:27:24.226112 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:27:24.226157 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":39,"thread_start_us":85,"threads_started":1}
I20260928 06:27:24.226466 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.221926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.31.214.65:36161)
0928 06:27:24.222321 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.222326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.222337 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:27:24.222381 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.222384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.222400 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.222695 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:24.222792 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.223923 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.225064 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.225808 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.226221 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.226289 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.226315 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.226367 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":130,"threads_started":1}
I20260928 06:27:24.226716 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.222252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.31.214.67:38993)
0928 06:27:24.222509 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.222514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.222525 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:27:24.222673 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.222676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.222694 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.222808 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:24.222912 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.223607 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.225181 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.225856 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.225920 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.226000 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.226511 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.226611 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
I20260928 06:27:24.226890 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.222035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:38993 (local address 127.0.0.1:57872)
0928 06:27:24.222466 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.222478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:24.222530 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:24.222803 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:24.222804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:24.222806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:24.222855 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.222857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.223718 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.223719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.225091 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.225094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.226031 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.226031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.226086 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.226088 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:27:24.226136 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:27:24.226170 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":72,"thread_start_us":74,"threads_started":1}
I20260928 06:27:24.227391 17424 tablet_service.cc:1467] Processing CreateTablet for tablet 82323a927f4e49f2b07bfb4f4239170a (DEFAULT_TABLE table=test-table [id=afcae781364b4f4a80336d385594dcdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:24.227396 17290 tablet_service.cc:1467] Processing CreateTablet for tablet 82323a927f4e49f2b07bfb4f4239170a (DEFAULT_TABLE table=test-table [id=afcae781364b4f4a80336d385594dcdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:24.227732 17290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82323a927f4e49f2b07bfb4f4239170a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:24.227737 17424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82323a927f4e49f2b07bfb4f4239170a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:24.230466 17641 tablet_bootstrap.cc:492] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721: Bootstrap starting.
I20260928 06:27:24.230499 17642 tablet_bootstrap.cc:492] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: Bootstrap starting.
I20260928 06:27:24.231189 17642 tablet_bootstrap.cc:654] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:24.231230 17641 tablet_bootstrap.cc:654] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:24.231232 17558 tablet_service.cc:1467] Processing CreateTablet for tablet 82323a927f4e49f2b07bfb4f4239170a (DEFAULT_TABLE table=test-table [id=afcae781364b4f4a80336d385594dcdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:24.231658 17558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82323a927f4e49f2b07bfb4f4239170a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:24.231766 17642 log.cc:824] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:24.231864 17641 log.cc:824] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:24.232568 17641 tablet_bootstrap.cc:492] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721: No bootstrap required, opened a new log
I20260928 06:27:24.232595 17642 tablet_bootstrap.cc:492] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: No bootstrap required, opened a new log
I20260928 06:27:24.232661 17642 ts_tablet_manager.cc:1397] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:27:24.232663 17641 ts_tablet_manager.cc:1397] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:27:24.234282 17641 raft_consensus.cc:348] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } }
I20260928 06:27:24.234411 17641 raft_consensus.cc:374] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:24.234431 17641 raft_consensus.cc:729] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c55841b24a14105a4cabb4b70cec721, State: Initialized, Role: FOLLOWER
I20260928 06:27:24.234551 17641 consensus_queue.cc:260] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } }
I20260928 06:27:24.234714 17642 raft_consensus.cc:348] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } }
I20260928 06:27:24.234859 17641 ts_tablet_manager.cc:1428] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:27:24.234879 17642 raft_consensus.cc:374] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:24.234887 17356 heartbeater.cc:503] Master 127.31.214.126:45663 was elected leader, sending a full tablet report...
I20260928 06:27:24.234918 17642 raft_consensus.cc:729] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1c50d800714b7c9205b9de5fc3aafb, State: Initialized, Role: FOLLOWER
I20260928 06:27:24.235030 17642 consensus_queue.cc:260] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } }
I20260928 06:27:24.235297 17642 ts_tablet_manager.cc:1428] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:27:24.235335 17490 heartbeater.cc:503] Master 127.31.214.126:45663 was elected leader, sending a full tablet report...
I20260928 06:27:24.235728 17647 tablet_bootstrap.cc:492] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: Bootstrap starting.
I20260928 06:27:24.236507 17647 tablet_bootstrap.cc:654] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:24.236991 17647 log.cc:824] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:24.237784 17647 tablet_bootstrap.cc:492] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: No bootstrap required, opened a new log
I20260928 06:27:24.237840 17647 ts_tablet_manager.cc:1397] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:27:24.239261 17647 raft_consensus.cc:348] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } }
I20260928 06:27:24.239396 17647 raft_consensus.cc:374] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:24.239418 17647 raft_consensus.cc:729] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b406edbf8b04391bc2ed41f1e9a12fb, State: Initialized, Role: FOLLOWER
I20260928 06:27:24.239564 17647 consensus_queue.cc:260] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } }
I20260928 06:27:24.239848 17647 ts_tablet_manager.cc:1428] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:27:24.239912 17624 heartbeater.cc:503] Master 127.31.214.126:45663 was elected leader, sending a full tablet report...
W20260928 06:27:24.272451 17491 tablet.cc:2369] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:27:24.334077 17357 tablet.cc:2369] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:27:24.421868 17646 raft_consensus.cc:482] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:24.422124 17646 raft_consensus.cc:504] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } }
I20260928 06:27:24.422590 17646 leader_election.cc:288] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c55841b24a14105a4cabb4b70cec721 (127.31.214.65:36161), 6b406edbf8b04391bc2ed41f1e9a12fb (127.31.214.67:38993)
I20260928 06:27:24.425552 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.422824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:38993 (local address 127.31.214.66:48867)
0928 06:27:24.422899 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.422916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:24.422984 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:24.423152 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:24.423154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:24.423156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:24.423236 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.423238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.423921 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.423922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.424677 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.424679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.425266 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.425266 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.425353 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.425356 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:24.425425 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:27:24.425470 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:27:24.425552 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.422940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:48867 (local address 127.31.214.67:38993)
0928 06:27:24.422999 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.423001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.423013 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:27:24.423049 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.423050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.423060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.423125 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:24.423316 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.423826 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.424779 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.425177 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.425267 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.425277 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.425432 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.425474 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:27:24.425921 17578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82323a927f4e49f2b07bfb4f4239170a" candidate_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" is_pre_election: true
I20260928 06:27:24.426160 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.422822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:36161 (local address 127.31.214.66:55041)
0928 06:27:24.423144 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.423154 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:24.423212 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:24.423413 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:24.423415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:24.423417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:24.423498 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.423503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.424256 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.424257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.425072 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.425074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.425834 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.425835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.425897 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.425901 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:24.426018 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:27:24.426071 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":15,"thread_start_us":242,"threads_started":1}
I20260928 06:27:24.426175 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.422975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:55041 (local address 127.31.214.65:36161)
0928 06:27:24.423024 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.423025 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.423237 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:27:24.423270 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.423271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.423280 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.423378 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:24.423623 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.424154 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.425239 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.425693 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.425779 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.425788 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.426053 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.426108 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:27:24.426192 17578 raft_consensus.cc:2454] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd1c50d800714b7c9205b9de5fc3aafb in term 0.
I20260928 06:27:24.426469 17378 leader_election.cc:302] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b406edbf8b04391bc2ed41f1e9a12fb, dd1c50d800714b7c9205b9de5fc3aafb; no voters:
I20260928 06:27:24.426555 17310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82323a927f4e49f2b07bfb4f4239170a" candidate_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c55841b24a14105a4cabb4b70cec721" is_pre_election: true
I20260928 06:27:24.426647 17646 raft_consensus.cc:2781] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:27:24.426723 17646 raft_consensus.cc:482] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:27:24.426739 17310 raft_consensus.cc:2454] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd1c50d800714b7c9205b9de5fc3aafb in term 0.
I20260928 06:27:24.426759 17646 raft_consensus.cc:3037] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:24.427621 17646 raft_consensus.cc:504] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } }
I20260928 06:27:24.427807 17646 leader_election.cc:288] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [CANDIDATE]: Term 1 election: Requested vote from peers 5c55841b24a14105a4cabb4b70cec721 (127.31.214.65:36161), 6b406edbf8b04391bc2ed41f1e9a12fb (127.31.214.67:38993)
I20260928 06:27:24.427956 17310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82323a927f4e49f2b07bfb4f4239170a" candidate_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c55841b24a14105a4cabb4b70cec721"
I20260928 06:27:24.428047 17310 raft_consensus.cc:3037] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:24.428083 17578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82323a927f4e49f2b07bfb4f4239170a" candidate_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb"
I20260928 06:27:24.428185 17578 raft_consensus.cc:3037] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:24.428910 17310 raft_consensus.cc:2454] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd1c50d800714b7c9205b9de5fc3aafb in term 1.
I20260928 06:27:24.428951 17578 raft_consensus.cc:2454] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd1c50d800714b7c9205b9de5fc3aafb in term 1.
I20260928 06:27:24.429173 17380 leader_election.cc:302] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c55841b24a14105a4cabb4b70cec721, dd1c50d800714b7c9205b9de5fc3aafb; no voters:
I20260928 06:27:24.429337 17646 raft_consensus.cc:2781] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:24.429499 17646 raft_consensus.cc:686] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 1 LEADER]: Becoming Leader. State: Replica: dd1c50d800714b7c9205b9de5fc3aafb, State: Running, Role: LEADER
I20260928 06:27:24.429613 17646 consensus_queue.cc:237] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } }
I20260928 06:27:24.430594 17164 catalog_manager.cc:5495] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb reported cstate change: term changed from 0 to 1, leader changed from <none> to dd1c50d800714b7c9205b9de5fc3aafb (127.31.214.66). New cstate: current_term: 1 leader_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd1c50d800714b7c9205b9de5fc3aafb" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 } health_report { overall_health: UNKNOWN } } }
W20260928 06:27:24.437572 17625 tablet.cc:2369] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:27:24.454202 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.451394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.31.214.66:37521)
0928 06:27:24.451479 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.451482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.451493 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:27:24.451548 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.451549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.451570 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.451663 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:24.451822 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.452390 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.453247 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.453475 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.453564 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.453764 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:24.453886 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:24.454026 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.454046 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.454070 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:27:24.456701 17578 raft_consensus.cc:1261] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [term 1 FOLLOWER]: Refusing update from remote peer dd1c50d800714b7c9205b9de5fc3aafb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:24.456701 17310 raft_consensus.cc:1261] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [term 1 FOLLOWER]: Refusing update from remote peer dd1c50d800714b7c9205b9de5fc3aafb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:24.457178 17651 consensus_queue.cc:1035] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b406edbf8b04391bc2ed41f1e9a12fb" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:24.457355 17646 consensus_queue.cc:1035] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c55841b24a14105a4cabb4b70cec721" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:24.460474 17655 mvcc.cc:205] Tried to move back new op lower bound from 7334202754891141120 to 7334202754784337920. Current Snapshot: MvccSnapshot[applied={T|T < 7334202754891141120}]
I20260928 06:27:24.461539 17656 mvcc.cc:205] Tried to move back new op lower bound from 7334202754891141120 to 7334202754784337920. Current Snapshot: MvccSnapshot[applied={T|T < 7334202754891141120 or (T in {7334202754891141120})}]
I20260928 06:27:24.463028 17165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:24.463146 17165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:24.464192 17165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8978b7a239dc432887f486fb48a556e3: Sending DeleteTablet for 3 replicas of tablet 82323a927f4e49f2b07bfb4f4239170a
I20260928 06:27:24.464632 17290 tablet_service.cc:1514] Processing DeleteTablet for tablet 82323a927f4e49f2b07bfb4f4239170a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:24 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:40444
I20260928 06:27:24.464666 17424 tablet_service.cc:1514] Processing DeleteTablet for tablet 82323a927f4e49f2b07bfb4f4239170a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:24 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:55798
I20260928 06:27:24.464931 17665 tablet_replica.cc:330] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721: stopping tablet replica
I20260928 06:27:24.465027 17558 tablet_service.cc:1514] Processing DeleteTablet for tablet 82323a927f4e49f2b07bfb4f4239170a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:24 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:57872
I20260928 06:27:24.465102 17666 tablet_replica.cc:330] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: stopping tablet replica
I20260928 06:27:24.465111 17665 raft_consensus.cc:2229] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:24.465214 17666 raft_consensus.cc:2229] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:27:24.465274 17665 raft_consensus.cc:2258] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:24.465317 17666 raft_consensus.cc:2258] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:24.465795 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 17227
I20260928 06:27:24.465860 17665 ts_tablet_manager.cc:1905] T 82323a927f4e49f2b07bfb4f4239170a P 5c55841b24a14105a4cabb4b70cec721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:24.468566 17666 ts_tablet_manager.cc:1905] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:24.470633 17666 ts_tablet_manager.cc:1918] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:24.470799 17666 log.cc:1196] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/82323a927f4e49f2b07bfb4f4239170a
I20260928 06:27:24.471199 17666 ts_tablet_manager.cc:1939] T 82323a927f4e49f2b07bfb4f4239170a P dd1c50d800714b7c9205b9de5fc3aafb: Deleting consensus metadata
I20260928 06:27:24.471777 17150 catalog_manager.cc:4841] TS dd1c50d800714b7c9205b9de5fc3aafb (127.31.214.66:37521): tablet 82323a927f4e49f2b07bfb4f4239170a (table test-table [id=afcae781364b4f4a80336d385594dcdc]) successfully deleted
I20260928 06:27:24.471939 17667 tablet_replica.cc:330] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: stopping tablet replica
W20260928 06:27:24.471930 17152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:36161 (error 108)
I20260928 06:27:24.472100 17667 raft_consensus.cc:2229] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:24.472344 17667 raft_consensus.cc:2258] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:24.472529 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.472387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:36161 (local address 127.0.0.1:40448)
0928 06:27:24.472449 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.472490 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:27:24.472662 17152 catalog_manager.cc:4584] TS 5c55841b24a14105a4cabb4b70cec721 (127.31.214.65:36161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82323a927f4e49f2b07bfb4f4239170a: Network error: Client connection negotiation failed: client connection to 127.31.214.65:36161: connect: Connection refused (error 111)
I20260928 06:27:24.472951 17667 ts_tablet_manager.cc:1905] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:24.473081 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 17361
I20260928 06:27:24.475325 17667 ts_tablet_manager.cc:1918] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:24.475423 17667 log.cc:1196] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/82323a927f4e49f2b07bfb4f4239170a
I20260928 06:27:24.475787 17667 ts_tablet_manager.cc:1939] T 82323a927f4e49f2b07bfb4f4239170a P 6b406edbf8b04391bc2ed41f1e9a12fb: Deleting consensus metadata
I20260928 06:27:24.476348 17150 catalog_manager.cc:4841] TS 6b406edbf8b04391bc2ed41f1e9a12fb (127.31.214.67:38993): tablet 82323a927f4e49f2b07bfb4f4239170a (table test-table [id=afcae781364b4f4a80336d385594dcdc]) successfully deleted
I20260928 06:27:24.479131 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 17495
I20260928 06:27:24.484280 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 17133
2026-09-28T06:27:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2163 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T06:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:24Z Disabled control of system clock
I20260928 06:27:24.514012 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38259
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:27:24.598297 17672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:24.598567 17672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:24.598590 17672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:24.599737 17672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:24.599772 17672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:24.599785 17672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:24.599799 17672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:24.599812 17672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:24.601205 17672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:38259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:24.601428 17672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:24.601676 17672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:24.606835 17680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:24.606840 17677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:24.606835 17678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:24.607209 17672 server_base.cc:1034] running on GCE node
I20260928 06:27:24.607514 17672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:24.607846 17672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:24.609038 17672 hybrid_clock.cc:648] HybridClock initialized: now 1790576844609027 us; error 36 us; skew 500 ppm
I20260928 06:27:24.610618 17672 webserver.cc:460] Webserver started at http://127.31.214.126:45807/ using document root <none> and password file <none>
I20260928 06:27:24.610903 17672 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:24.610975 17672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:24.611135 17672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:24.612274 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "e4ca67d4c3dc4236acef48e682d8bad6"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "3d902ad284b7b7c397806dba1da1c3f2"
server_key_iv: "7735d4a6955cb2cb77cf9813534e11ae"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.612661 17672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "e4ca67d4c3dc4236acef48e682d8bad6"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "3d902ad284b7b7c397806dba1da1c3f2"
server_key_iv: "7735d4a6955cb2cb77cf9813534e11ae"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.614151 17672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:27:24.615175 17686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.615458 17672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:24.615615 17672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "e4ca67d4c3dc4236acef48e682d8bad6"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "3d902ad284b7b7c397806dba1da1c3f2"
server_key_iv: "7735d4a6955cb2cb77cf9813534e11ae"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.615711 17672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:24.629070 17672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:24.629475 17672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:24.629664 17672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:24.634196 17672 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:38259
I20260928 06:27:24.634313 17738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:38259 every 8 connection(s)
I20260928 06:27:24.634763 17672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:24.635922 17739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:24.638463 17739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6: Bootstrap starting.
I20260928 06:27:24.639104 17739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:24.639106 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 17672
I20260928 06:27:24.639236 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:24.639447 17739 log.cc:824] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:24.639497 32601 external_mini_cluster.cc:1428] Setting key 17ba00f8ae9d9de9bdaa4790378be9d8
I20260928 06:27:24.640344 17739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6: No bootstrap required, opened a new log
I20260928 06:27:24.642800 17739 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca67d4c3dc4236acef48e682d8bad6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38259 } }
I20260928 06:27:24.642977 17739 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:24.643041 17739 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ca67d4c3dc4236acef48e682d8bad6, State: Initialized, Role: FOLLOWER
I20260928 06:27:24.643219 17739 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca67d4c3dc4236acef48e682d8bad6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38259 } }
I20260928 06:27:24.643313 17739 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:24.643344 17739 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:24.643395 17739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:24.644305 17739 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca67d4c3dc4236acef48e682d8bad6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38259 } }
I20260928 06:27:24.644435 17739 leader_election.cc:302] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4ca67d4c3dc4236acef48e682d8bad6; no voters:
I20260928 06:27:24.644680 17739 leader_election.cc:288] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:24.644713 17744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:24.644893 17744 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [term 1 LEADER]: Becoming Leader. State: Replica: e4ca67d4c3dc4236acef48e682d8bad6, State: Running, Role: LEADER
I20260928 06:27:24.645032 17744 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca67d4c3dc4236acef48e682d8bad6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38259 } }
I20260928 06:27:24.645077 17739 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:24.645453 17745 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4ca67d4c3dc4236acef48e682d8bad6. Latest consensus state: current_term: 1 leader_uuid: "e4ca67d4c3dc4236acef48e682d8bad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca67d4c3dc4236acef48e682d8bad6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38259 } } }
I20260928 06:27:24.645545 17745 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:24.645727 17746 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4ca67d4c3dc4236acef48e682d8bad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ca67d4c3dc4236acef48e682d8bad6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38259 } } }
I20260928 06:27:24.645809 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.640134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.31.214.126:38259)
0928 06:27:24.640380 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.640388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.640412 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:27:24.640897 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.640905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.640969 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.641131 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:24.641309 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.642291 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.643113 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.643820 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.643970 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.644785 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:24.644794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:24.644810 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:24.645047 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:24.645096 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:24.645111 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:24.645404 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.645437 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.645525 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
I20260928 06:27:24.645812 17748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:24.645879 17746 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:24.646931 17748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:24.648478 17748 catalog_manager.cc:1347] Generated new cluster ID: d059ea8265034b06adedd1967244114b
I20260928 06:27:24.648586 17748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:24.663512 17748 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:24.664299 17748 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:24.668279 17748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6: Generated new TSK 0
I20260928 06:27:24.668577 17748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:24.676716 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38259
--builtin_ntp_servers=127.31.214.84:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:27:24.763406 17763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:24.763651 17763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:24.763675 17763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:24.764878 17763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:24.764920 17763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:24.764947 17763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:27:24.766325 17763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:24.766564 17763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:24.766904 17763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:24.772356 17768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:24.772574 17771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:24.772333 17769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:24.772805 17763 server_base.cc:1034] running on GCE node
I20260928 06:27:24.773013 17763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:24.773345 17763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:24.774513 17763 hybrid_clock.cc:648] HybridClock initialized: now 1790576844774500 us; error 32 us; skew 500 ppm
I20260928 06:27:24.775959 17763 webserver.cc:460] Webserver started at http://127.31.214.65:43255/ using document root <none> and password file <none>
I20260928 06:27:24.776209 17763 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:24.776279 17763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:24.776433 17763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:24.777570 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "dd228668c5e34cc0a8b26600f068d3a2"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "10566ae0f1b2f9ea64753595c9b9c47a"
server_key_iv: "43ff009d299bcba13e9c7a9d7ff7246e"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.777990 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "dd228668c5e34cc0a8b26600f068d3a2"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "10566ae0f1b2f9ea64753595c9b9c47a"
server_key_iv: "43ff009d299bcba13e9c7a9d7ff7246e"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.779395 17763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:24.780202 17777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.780472 17763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:24.780588 17763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "dd228668c5e34cc0a8b26600f068d3a2"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "10566ae0f1b2f9ea64753595c9b9c47a"
server_key_iv: "43ff009d299bcba13e9c7a9d7ff7246e"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.780669 17763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:24.800601 17763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:24.800930 17763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:24.801046 17763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:24.801276 17763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:24.801657 17763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:24.801694 17763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.801720 17763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:24.801734 17763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.808737 17763 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:46283
I20260928 06:27:24.808847 17890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:46283 every 8 connection(s)
I20260928 06:27:24.809247 17763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:27:24.812103 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 17763
I20260928 06:27:24.812192 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:27:24.812526 32601 external_mini_cluster.cc:1428] Setting key 3a7c40cadb98d3c04e5f1fbfe393ee50
I20260928 06:27:24.813602 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38259
--builtin_ntp_servers=127.31.214.84:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:27:24.815280 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.810203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:57707 (local address 127.31.214.126:38259)
0928 06:27:24.810275 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.810278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.810539 (+ 261us) server_negotiation.cc:407] Connection header received
0928 06:27:24.811099 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.811100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.811112 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.811170 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:24.811922 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.812608 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.813627 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.813917 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.814038 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.815020 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:24.815022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:24.815022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:24.815027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:24.815046 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:24.815051 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:24.815104 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.815188 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.815217 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:27:24.815644 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.810211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38259 (local address 127.31.214.65:57707)
0928 06:27:24.810402 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.810438 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:24.810976 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:24.811250 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:24.811254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:24.811263 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:24.811772 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.811776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.812703 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.812704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.813531 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.813534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.814051 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.814052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.814196 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.814846 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:27:24.814847 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:24.814896 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:27:24.814934 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:24.815107 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:24.815111 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:24.815114 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:24.815192 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:27:24.815313 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260928 06:27:24.816278 17891 heartbeater.cc:347] Connected to a master server at 127.31.214.126:38259
I20260928 06:27:24.816385 17891 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:24.816614 17891 heartbeater.cc:511] Master 127.31.214.126:38259 requested a full tablet report, sending...
I20260928 06:27:24.817298 17703 ts_manager.cc:194] Registered new tserver with Master: dd228668c5e34cc0a8b26600f068d3a2 (127.31.214.65:46283)
I20260928 06:27:24.818118 17703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.65:57707
W20260928 06:27:24.899616 17894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:24.899840 17894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:24.899865 17894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:24.900995 17894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:24.901028 17894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:24.901053 17894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:27:24.902431 17894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:24.902638 17894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:24.902907 17894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:24.908407 17902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:24.908456 17900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:24.908550 17899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:24.908635 17894 server_base.cc:1034] running on GCE node
I20260928 06:27:24.909013 17894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:24.909291 17894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:24.910490 17894 hybrid_clock.cc:648] HybridClock initialized: now 1790576844910453 us; error 37 us; skew 500 ppm
I20260928 06:27:24.911913 17894 webserver.cc:460] Webserver started at http://127.31.214.66:45725/ using document root <none> and password file <none>
I20260928 06:27:24.912248 17894 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:24.912322 17894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:24.912483 17894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:24.913578 17894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "7d299e8e433943899eae03149831741a"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "fc3137d918f28e0e2acdbbf23ee553ef"
server_key_iv: "0bdd4b7dee77f173d8ba27bc2fc28201"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.914005 17894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "7d299e8e433943899eae03149831741a"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "fc3137d918f28e0e2acdbbf23ee553ef"
server_key_iv: "0bdd4b7dee77f173d8ba27bc2fc28201"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.915452 17894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:24.916332 17908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.916544 17894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:24.916657 17894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "7d299e8e433943899eae03149831741a"
format_stamp: "Formatted at 2026-09-28 06:27:24 on dist-test-slave-q03f"
server_key: "fc3137d918f28e0e2acdbbf23ee553ef"
server_key_iv: "0bdd4b7dee77f173d8ba27bc2fc28201"
server_key_version: "encryptionkey@0"
I20260928 06:27:24.916749 17894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:24.948526 17894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:24.948989 17894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:24.949236 17894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:24.949522 17894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:24.949921 17894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:24.950021 17894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.950064 17894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:24.950150 17894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:24.956661 17894 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:45727
I20260928 06:27:24.956753 18021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:45727 every 8 connection(s)
I20260928 06:27:24.957144 17894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:27:24.959985 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 17894
I20260928 06:27:24.960108 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:27:24.960336 32601 external_mini_cluster.cc:1428] Setting key d61b1df332d8a42400e791d814cf79c5
I20260928 06:27:24.961547 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38259
--builtin_ntp_servers=127.31.214.84:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:27:24.964138 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.958087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:53679 (local address 127.31.214.126:38259)
0928 06:27:24.958158 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:24.958161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:24.958380 (+ 219us) server_negotiation.cc:407] Connection header received
0928 06:27:24.959064 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:24.959066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:24.959078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:24.959147 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:24.959972 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.960790 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.962253 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.962620 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.962709 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.963791 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:24.963793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:24.963793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:24.963801 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:24.963822 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:24.963828 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:24.963905 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:24.963988 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:27:24.964023 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:27:24.964409 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:24.957978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38259 (local address 127.31.214.66:53679)
0928 06:27:24.958219 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:24.958261 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:24.958924 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:24.959247 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:24.959251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:24.959262 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:24.959845 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.959851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.960900 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.960901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.962155 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:24.962159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:24.962749 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:24.962750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:24.962890 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:24.963621 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:27:24.963622 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:24.963658 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:27:24.963686 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:24.963907 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:24.963910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:24.963913 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:24.963965 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:27:24.964061 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260928 06:27:24.965014 18022 heartbeater.cc:347] Connected to a master server at 127.31.214.126:38259
I20260928 06:27:24.965148 18022 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:24.965359 18022 heartbeater.cc:511] Master 127.31.214.126:38259 requested a full tablet report, sending...
I20260928 06:27:24.965858 17703 ts_manager.cc:194] Registered new tserver with Master: 7d299e8e433943899eae03149831741a (127.31.214.66:45727)
I20260928 06:27:24.966570 17703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.66:53679
W20260928 06:27:25.050209 18025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:25.050427 18025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:25.050450 18025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:25.051569 18025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:25.051606 18025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:25.051631 18025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:27:25.052888 18025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:38259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:25.053084 18025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:25.053330 18025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:25.058861 18031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:25.058883 18033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:25.059075 18025 server_base.cc:1034] running on GCE node
W20260928 06:27:25.059139 18030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:25.059405 18025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:25.059715 18025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:25.060887 18025 hybrid_clock.cc:648] HybridClock initialized: now 1790576845060884 us; error 62 us; skew 500 ppm
I20260928 06:27:25.062348 18025 webserver.cc:460] Webserver started at http://127.31.214.67:35245/ using document root <none> and password file <none>
I20260928 06:27:25.062650 18025 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:25.062729 18025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:25.062897 18025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:25.064038 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "c46edaee9b974e8cb15a6a2089871ab1"
format_stamp: "Formatted at 2026-09-28 06:27:25 on dist-test-slave-q03f"
server_key: "0c83b8a7050acb37815776310d8ae00d"
server_key_iv: "bbc115982bad45186250916080a44d97"
server_key_version: "encryptionkey@0"
I20260928 06:27:25.064426 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "c46edaee9b974e8cb15a6a2089871ab1"
format_stamp: "Formatted at 2026-09-28 06:27:25 on dist-test-slave-q03f"
server_key: "0c83b8a7050acb37815776310d8ae00d"
server_key_iv: "bbc115982bad45186250916080a44d97"
server_key_version: "encryptionkey@0"
I20260928 06:27:25.065855 18025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:27:25.066774 18039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:25.067020 18025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:25.067137 18025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "c46edaee9b974e8cb15a6a2089871ab1"
format_stamp: "Formatted at 2026-09-28 06:27:25 on dist-test-slave-q03f"
server_key: "0c83b8a7050acb37815776310d8ae00d"
server_key_iv: "bbc115982bad45186250916080a44d97"
server_key_version: "encryptionkey@0"
I20260928 06:27:25.067216 18025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:25.096802 18025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:25.097174 18025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:25.097344 18025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:25.097626 18025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:25.098047 18025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:25.098104 18025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:25.098150 18025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:25.098177 18025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:25.104377 18025 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:40895
I20260928 06:27:25.104399 18152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:40895 every 8 connection(s)
I20260928 06:27:25.104804 18025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:27:25.108341 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 18025
I20260928 06:27:25.108467 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:27:25.108652 32601 external_mini_cluster.cc:1428] Setting key 26a9928d2f20e11dab7d5c1b27a0ca27
I20260928 06:27:25.110728 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.105692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:47579 (local address 127.31.214.126:38259)
0928 06:27:25.105757 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:25.105760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:25.106006 (+ 246us) server_negotiation.cc:407] Connection header received
0928 06:27:25.106532 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:25.106533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:25.106545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:25.106618 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:25.107288 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.107789 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.109056 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.109371 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.109442 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.110490 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:25.110491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:25.110491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:25.110496 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:25.110512 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:25.110516 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:25.110574 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:25.110643 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:27:25.110667 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:27:25.110934 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.105608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:38259 (local address 127.31.214.67:47579)
0928 06:27:25.105811 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:25.105853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:25.106428 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:25.106677 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:25.106680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:25.106689 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:25.107193 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.107198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.107924 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.107925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.108934 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.108938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.109458 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.109459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.109682 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.110342 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:27:25.110343 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:25.110375 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:27:25.110400 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:25.110577 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:25.110580 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:25.110583 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:25.110631 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:27:25.110716 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260928 06:27:25.111464 18153 heartbeater.cc:347] Connected to a master server at 127.31.214.126:38259
I20260928 06:27:25.111562 18153 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:25.111714 18153 heartbeater.cc:511] Master 127.31.214.126:38259 requested a full tablet report, sending...
I20260928 06:27:25.112151 17703 ts_manager.cc:194] Registered new tserver with Master: c46edaee9b974e8cb15a6a2089871ab1 (127.31.214.67:40895)
I20260928 06:27:25.112545 17703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.67:47579
I20260928 06:27:25.121025 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:27:25.126292 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.122862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.31.214.126:38259)
0928 06:27:25.123004 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:25.123006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:25.123100 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:27:25.123160 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:25.123161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:25.123170 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:25.123219 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:25.123358 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.123861 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.124867 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.125139 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.125203 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.125871 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:25.125872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:25.125872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:25.125876 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:25.125889 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:25.125893 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:25.126091 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:25.126149 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:27:25.126172 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:27:25.127948 17703 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:27:25.129767 17703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:27:25.137013 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.132319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46283 (local address 127.0.0.1:47854)
0928 06:27:25.132849 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:25.132863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:25.132926 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:25.133206 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:25.133207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:25.133208 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:25.133248 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.133249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.134593 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.134594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.135856 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.135859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.136718 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.136719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.136781 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.136784 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:25.136854 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:27:25.136927 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":6,"thread_start_us":76,"threads_started":1}
I20260928 06:27:25.137055 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.132609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.31.214.67:40895)
0928 06:27:25.132819 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:25.132824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:25.133087 (+ 263us) server_negotiation.cc:407] Connection header received
0928 06:27:25.133135 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:25.133138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:25.133153 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:25.133228 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:25.133476 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.134679 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.135787 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.136562 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.136645 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.136748 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:25.136907 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 06:27:25.136956 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260928 06:27:25.137305 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.132494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:40895 (local address 127.0.0.1:33894)
0928 06:27:25.132932 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:25.132948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:25.133015 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:25.133333 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:25.133334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:25.133336 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:25.133384 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.133386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.134857 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.134858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.135691 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.135693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.136651 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.136651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.136750 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.136768 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 06:27:25.136859 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:27:25.137258 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":359,"thread_start_us":79,"threads_started":1}
I20260928 06:27:25.137005 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.132488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.31.214.65:46283)
0928 06:27:25.132728 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:25.132734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:25.132995 (+ 261us) server_negotiation.cc:407] Connection header received
0928 06:27:25.133046 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:25.133049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:25.133065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:25.133139 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:25.133321 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.134505 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.135962 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.136644 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.136708 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.136782 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:25.136875 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:27:25.136915 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
I20260928 06:27:25.138031 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.132264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:45727 (local address 127.0.0.1:49744)
0928 06:27:25.132638 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:25.132690 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:25.132767 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:25.133141 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:25.133144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:25.133146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:25.133216 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.133217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.134369 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.134370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.135856 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.135859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.137758 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.137759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.137824 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.137829 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:27:25.137881 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:27:25.137921 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":30,"thread_start_us":80,"threads_started":1}
I20260928 06:27:25.138353 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.132410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.31.214.66:45727)
0928 06:27:25.132963 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:25.132968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:25.132980 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:27:25.133028 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:25.133031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:25.133050 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:25.133137 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:25.133303 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.134277 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.136818 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.137561 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.137628 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.137724 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:25.138237 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 06:27:25.138281 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":109,"threads_started":1}
I20260928 06:27:25.138315 18087 tablet_service.cc:1467] Processing CreateTablet for tablet 021b3128a36848a4873531054d26d4ca (DEFAULT_TABLE table=test-table [id=d0dab9b5c1df40d1ad1389e02f3fe39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:25.138451 17825 tablet_service.cc:1467] Processing CreateTablet for tablet 021b3128a36848a4873531054d26d4ca (DEFAULT_TABLE table=test-table [id=d0dab9b5c1df40d1ad1389e02f3fe39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:25.138754 18087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021b3128a36848a4873531054d26d4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:25.138811 17825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021b3128a36848a4873531054d26d4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:25.142639 18168 tablet_bootstrap.cc:492] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: Bootstrap starting.
I20260928 06:27:25.142691 17956 tablet_service.cc:1467] Processing CreateTablet for tablet 021b3128a36848a4873531054d26d4ca (DEFAULT_TABLE table=test-table [id=d0dab9b5c1df40d1ad1389e02f3fe39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:25.142907 18167 tablet_bootstrap.cc:492] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2: Bootstrap starting.
I20260928 06:27:25.143115 17956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021b3128a36848a4873531054d26d4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:25.143664 18167 tablet_bootstrap.cc:654] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:25.144155 18167 log.cc:824] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:25.145061 18167 tablet_bootstrap.cc:492] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2: No bootstrap required, opened a new log
I20260928 06:27:25.145141 18168 tablet_bootstrap.cc:654] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:25.145152 18167 ts_tablet_manager.cc:1397] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:27:25.145514 18168 log.cc:824] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:25.147401 18171 tablet_bootstrap.cc:492] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: Bootstrap starting.
I20260928 06:27:25.147743 18167 raft_consensus.cc:348] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } }
I20260928 06:27:25.147957 18167 raft_consensus.cc:374] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:25.147943 18168 tablet_bootstrap.cc:492] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: No bootstrap required, opened a new log
I20260928 06:27:25.147993 18167 raft_consensus.cc:729] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd228668c5e34cc0a8b26600f068d3a2, State: Initialized, Role: FOLLOWER
I20260928 06:27:25.148036 18168 ts_tablet_manager.cc:1397] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 06:27:25.148144 18171 tablet_bootstrap.cc:654] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:25.148175 18167 consensus_queue.cc:260] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } }
I20260928 06:27:25.148487 18167 ts_tablet_manager.cc:1428] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:27:25.148499 18171 log.cc:824] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:25.148540 17891 heartbeater.cc:503] Master 127.31.214.126:38259 was elected leader, sending a full tablet report...
I20260928 06:27:25.149230 18171 tablet_bootstrap.cc:492] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: No bootstrap required, opened a new log
I20260928 06:27:25.149302 18171 ts_tablet_manager.cc:1397] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:27:25.150892 18168 raft_consensus.cc:348] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } }
I20260928 06:27:25.151060 18168 raft_consensus.cc:374] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:25.151111 18168 raft_consensus.cc:729] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46edaee9b974e8cb15a6a2089871ab1, State: Initialized, Role: FOLLOWER
I20260928 06:27:25.151247 18168 consensus_queue.cc:260] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } }
I20260928 06:27:25.151530 18168 ts_tablet_manager.cc:1428] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:27:25.151583 18153 heartbeater.cc:503] Master 127.31.214.126:38259 was elected leader, sending a full tablet report...
I20260928 06:27:25.151584 18171 raft_consensus.cc:348] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } }
I20260928 06:27:25.151768 18171 raft_consensus.cc:374] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:25.151808 18171 raft_consensus.cc:729] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d299e8e433943899eae03149831741a, State: Initialized, Role: FOLLOWER
I20260928 06:27:25.151927 18171 consensus_queue.cc:260] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } }
I20260928 06:27:25.152215 18171 ts_tablet_manager.cc:1428] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:27:25.152392 18022 heartbeater.cc:503] Master 127.31.214.126:38259 was elected leader, sending a full tablet report...
W20260928 06:27:25.207839 18023 tablet.cc:2369] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:27:25.221869 18174 raft_consensus.cc:482] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:25.222137 18174 raft_consensus.cc:504] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } }
I20260928 06:27:25.222636 18174 leader_election.cc:288] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d299e8e433943899eae03149831741a (127.31.214.66:45727), dd228668c5e34cc0a8b26600f068d3a2 (127.31.214.65:46283)
I20260928 06:27:25.226828 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.222849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46283 (local address 127.31.214.67:45361)
0928 06:27:25.222904 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:25.222917 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:25.222977 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:25.223191 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:25.223192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:25.223195 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:25.223296 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.223298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.224263 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.224264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.225591 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.225594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.226526 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.226527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.226614 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.226620 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:27:25.226695 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:27:25.226746 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":37}
I20260928 06:27:25.226917 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.222930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:45361 (local address 127.31.214.65:46283)
0928 06:27:25.223010 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:25.223013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:25.223025 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:27:25.223059 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:25.223060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:25.223073 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:25.223137 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:25.223380 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.224162 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.225732 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.226396 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.226522 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.226534 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:25.226788 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 06:27:25.226841 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:27:25.227044 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.222996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:35401 (local address 127.31.214.66:45727)
0928 06:27:25.223041 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:25.223044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:25.223177 (+ 133us) server_negotiation.cc:407] Connection header received
0928 06:27:25.223210 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:25.223211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:25.223222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:25.223276 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:25.223471 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.224257 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.225731 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.226505 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.226578 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.226586 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:25.226873 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:27:25.226932 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:27:25.227288 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.222853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:45727 (local address 127.31.214.67:35401)
0928 06:27:25.223095 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:25.223104 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:25.223161 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:25.223303 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:25.223303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:25.223305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:25.223386 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.223387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.224339 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.224340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.225616 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:25.225618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.226632 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.226633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:25.226766 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.226771 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:27:25.227146 (+ 375us) client_negotiation.cc:240] Negotiation successful
0928 06:27:25.227198 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":44,"thread_start_us":145,"threads_started":1}
I20260928 06:27:25.227263 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021b3128a36848a4873531054d26d4ca" candidate_uuid: "c46edaee9b974e8cb15a6a2089871ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd228668c5e34cc0a8b26600f068d3a2" is_pre_election: true
I20260928 06:27:25.227486 17845 raft_consensus.cc:2454] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c46edaee9b974e8cb15a6a2089871ab1 in term 0.
I20260928 06:27:25.227684 17976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021b3128a36848a4873531054d26d4ca" candidate_uuid: "c46edaee9b974e8cb15a6a2089871ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d299e8e433943899eae03149831741a" is_pre_election: true
I20260928 06:27:25.227749 18043 leader_election.cc:302] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c46edaee9b974e8cb15a6a2089871ab1, dd228668c5e34cc0a8b26600f068d3a2; no voters:
I20260928 06:27:25.227852 17976 raft_consensus.cc:2454] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c46edaee9b974e8cb15a6a2089871ab1 in term 0.
I20260928 06:27:25.227880 18174 raft_consensus.cc:2781] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:27:25.227932 18174 raft_consensus.cc:482] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:27:25.227965 18174 raft_consensus.cc:3037] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:25.228747 18174 raft_consensus.cc:504] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } }
I20260928 06:27:25.228897 18174 leader_election.cc:288] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [CANDIDATE]: Term 1 election: Requested vote from peers 7d299e8e433943899eae03149831741a (127.31.214.66:45727), dd228668c5e34cc0a8b26600f068d3a2 (127.31.214.65:46283)
I20260928 06:27:25.229064 17976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021b3128a36848a4873531054d26d4ca" candidate_uuid: "c46edaee9b974e8cb15a6a2089871ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d299e8e433943899eae03149831741a"
I20260928 06:27:25.229147 17976 raft_consensus.cc:3037] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:25.229178 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021b3128a36848a4873531054d26d4ca" candidate_uuid: "c46edaee9b974e8cb15a6a2089871ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd228668c5e34cc0a8b26600f068d3a2"
I20260928 06:27:25.229251 17845 raft_consensus.cc:3037] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:25.230168 17976 raft_consensus.cc:2454] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c46edaee9b974e8cb15a6a2089871ab1 in term 1.
I20260928 06:27:25.230170 17845 raft_consensus.cc:2454] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c46edaee9b974e8cb15a6a2089871ab1 in term 1.
I20260928 06:27:25.230422 18043 leader_election.cc:302] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c46edaee9b974e8cb15a6a2089871ab1, dd228668c5e34cc0a8b26600f068d3a2; no voters:
I20260928 06:27:25.230525 18174 raft_consensus.cc:2781] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:25.230700 18174 raft_consensus.cc:686] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 1 LEADER]: Becoming Leader. State: Replica: c46edaee9b974e8cb15a6a2089871ab1, State: Running, Role: LEADER
I20260928 06:27:25.230801 18174 consensus_queue.cc:237] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } }
I20260928 06:27:25.231671 17703 catalog_manager.cc:5495] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c46edaee9b974e8cb15a6a2089871ab1 (127.31.214.67). New cstate: current_term: 1 leader_uuid: "c46edaee9b974e8cb15a6a2089871ab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c46edaee9b974e8cb15a6a2089871ab1" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 40895 } health_report { overall_health: HEALTHY } } }
I20260928 06:27:25.255694 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.252924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.31.214.67:40895)
0928 06:27:25.253027 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:25.253029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:25.253040 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:27:25.253098 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:25.253099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:25.253121 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:25.253196 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:25.253418 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.254033 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.254867 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:25.255119 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:25.255174 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:25.255323 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:25.255437 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:25.255504 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:25.255538 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:27:25.255557 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:27:25.257872 17845 raft_consensus.cc:1261] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2 [term 1 FOLLOWER]: Refusing update from remote peer c46edaee9b974e8cb15a6a2089871ab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:25.257894 17976 raft_consensus.cc:1261] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [term 1 FOLLOWER]: Refusing update from remote peer c46edaee9b974e8cb15a6a2089871ab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:25.258358 18177 consensus_queue.cc:1035] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd228668c5e34cc0a8b26600f068d3a2" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:25.258486 18174 consensus_queue.cc:1035] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d299e8e433943899eae03149831741a" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:25.261344 18180 mvcc.cc:205] Tried to move back new op lower bound from 7334202758173204480 to 7334202758066192384. Current Snapshot: MvccSnapshot[applied={T|T < 7334202758173204480}]
I20260928 06:27:25.262041 18182 mvcc.cc:205] Tried to move back new op lower bound from 7334202758173204480 to 7334202758066192384. Current Snapshot: MvccSnapshot[applied={T|T < 7334202758173204480}]
I20260928 06:27:25.264456 17703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:25.264565 17703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:25.265434 17703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4ca67d4c3dc4236acef48e682d8bad6: Sending DeleteTablet for 3 replicas of tablet 021b3128a36848a4873531054d26d4ca
I20260928 06:27:25.265828 17825 tablet_service.cc:1514] Processing DeleteTablet for tablet 021b3128a36848a4873531054d26d4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:25 UTC) from {username='slave'} at 127.0.0.1:47854
I20260928 06:27:25.265890 18087 tablet_service.cc:1514] Processing DeleteTablet for tablet 021b3128a36848a4873531054d26d4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:25 UTC) from {username='slave'} at 127.0.0.1:33894
I20260928 06:27:25.265969 17956 tablet_service.cc:1514] Processing DeleteTablet for tablet 021b3128a36848a4873531054d26d4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:25 UTC) from {username='slave'} at 127.0.0.1:49744
I20260928 06:27:25.266202 18190 tablet_replica.cc:330] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2: stopping tablet replica
I20260928 06:27:25.266209 18192 tablet_replica.cc:330] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: stopping tablet replica
I20260928 06:27:25.266315 18192 raft_consensus.cc:2229] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:25.266341 18190 raft_consensus.cc:2229] T 021b3128a36848a4873531054d26d4ca P dd228668c5e34cc0a8b26600f068d3a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:25.266399 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 17763
I20260928 06:27:25.266456 18192 raft_consensus.cc:2258] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:25.268702 18192 ts_tablet_manager.cc:1905] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:25.270548 18192 ts_tablet_manager.cc:1918] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:25.270663 18192 log.cc:1196] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/021b3128a36848a4873531054d26d4ca
I20260928 06:27:25.270960 18192 ts_tablet_manager.cc:1939] T 021b3128a36848a4873531054d26d4ca P 7d299e8e433943899eae03149831741a: Deleting consensus metadata
I20260928 06:27:25.271490 17689 catalog_manager.cc:4841] TS 7d299e8e433943899eae03149831741a (127.31.214.66:45727): tablet 021b3128a36848a4873531054d26d4ca (table test-table [id=d0dab9b5c1df40d1ad1389e02f3fe39b]) successfully deleted
W20260928 06:27:25.271998 17690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:46283 (error 108)
I20260928 06:27:25.272193 18191 tablet_replica.cc:330] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: stopping tablet replica
I20260928 06:27:25.272312 18191 raft_consensus.cc:2229] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:27:25.272411 18191 raft_consensus.cc:2258] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:25.272596 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 17894
I20260928 06:27:25.273247 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:25.272684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46283 (local address 127.0.0.1:47856)
0928 06:27:25.273156 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:25.273202 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449}
I20260928 06:27:25.273579 18191 ts_tablet_manager.cc:1905] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:27:25.274834 17690 catalog_manager.cc:4584] TS dd228668c5e34cc0a8b26600f068d3a2 (127.31.214.65:46283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 021b3128a36848a4873531054d26d4ca: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46283: connect: Connection refused (error 111)
I20260928 06:27:25.275295 18191 ts_tablet_manager.cc:1918] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:25.275391 18191 log.cc:1196] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/021b3128a36848a4873531054d26d4ca
I20260928 06:27:25.275679 18191 ts_tablet_manager.cc:1939] T 021b3128a36848a4873531054d26d4ca P c46edaee9b974e8cb15a6a2089871ab1: Deleting consensus metadata
I20260928 06:27:25.276126 17689 catalog_manager.cc:4841] TS c46edaee9b974e8cb15a6a2089871ab1 (127.31.214.67:40895): tablet 021b3128a36848a4873531054d26d4ca (table test-table [id=d0dab9b5c1df40d1ad1389e02f3fe39b]) successfully deleted
I20260928 06:27:25.279140 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 18025
I20260928 06:27:25.284049 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 17672
2026-09-28T06:27:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (797 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:25 dist-test-slave-q03f krb5kdc[18196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:25 dist-test-slave-q03f krb5kdc[18196](info): set up 2 sockets
Sep 28 06:27:25 dist-test-slave-q03f krb5kdc[18196](info): commencing operation
krb5kdc: starting...
W20260928 06:27:26.333829 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:26 dist-test-slave-q03f krb5kdc[18196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:26.461830 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45507
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:45507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:26.544317 18209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:26.544577 18209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:26.544616 18209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:26.545791 18209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:26.545862 18209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:26.545885 18209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:26.545907 18209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:26.545933 18209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:26.547542 18209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:45507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:26.547827 18209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:26.548076 18209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:26.553484 18214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:26.553465 18217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:26.553481 18215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:26.553844 18209 server_base.cc:1034] running on GCE node
I20260928 06:27:26.554138 18209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:26.554452 18209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:26.555627 18209 hybrid_clock.cc:648] HybridClock initialized: now 1790576846555615 us; error 47 us; skew 500 ppm
Sep 28 06:27:26 dist-test-slave-q03f krb5kdc[18196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:26.558328 18209 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:26.558854 18209 webserver.cc:460] Webserver started at http://127.31.214.126:35137/ using document root <none> and password file <none>
I20260928 06:27:26.559116 18209 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:26.559188 18209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:26.559352 18209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:26.560467 18209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "26124623207046ae98a0a9368fff515b"
format_stamp: "Formatted at 2026-09-28 06:27:26 on dist-test-slave-q03f"
server_key: "c79b3e967b43b667dcb712c65308a153"
server_key_iv: "e80fdc34286ebc952d6b04614e84ebe0"
server_key_version: "encryptionkey@0"
I20260928 06:27:26.560897 18209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "26124623207046ae98a0a9368fff515b"
format_stamp: "Formatted at 2026-09-28 06:27:26 on dist-test-slave-q03f"
server_key: "c79b3e967b43b667dcb712c65308a153"
server_key_iv: "e80fdc34286ebc952d6b04614e84ebe0"
server_key_version: "encryptionkey@0"
I20260928 06:27:26.562354 18209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:27:26.563196 18224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:26.563372 18209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:27:26.563477 18209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "26124623207046ae98a0a9368fff515b"
format_stamp: "Formatted at 2026-09-28 06:27:26 on dist-test-slave-q03f"
server_key: "c79b3e967b43b667dcb712c65308a153"
server_key_iv: "e80fdc34286ebc952d6b04614e84ebe0"
server_key_version: "encryptionkey@0"
I20260928 06:27:26.563575 18209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:26.576375 18209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:26.578636 18209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:26.578807 18209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:26.583053 18209 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:45507
I20260928 06:27:26.583096 18276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:45507 every 8 connection(s)
I20260928 06:27:26.583483 18209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:26.584376 18277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:26.586721 18277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b: Bootstrap starting.
I20260928 06:27:26.587159 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 18209
I20260928 06:27:26.587296 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:26.587447 18277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:26.587512 32601 external_mini_cluster.cc:1428] Setting key edb114bc51699c4df69d38ec79228b79
I20260928 06:27:26.587750 18277 log.cc:824] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:26.588521 18277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b: No bootstrap required, opened a new log
I20260928 06:27:26.590291 18277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26124623207046ae98a0a9368fff515b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45507 } }
I20260928 06:27:26.590467 18277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:26.590507 18277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26124623207046ae98a0a9368fff515b, State: Initialized, Role: FOLLOWER
I20260928 06:27:26.590694 18277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26124623207046ae98a0a9368fff515b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45507 } }
I20260928 06:27:26.590794 18277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:26.590833 18277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:26.590888 18277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:26.591902 18277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26124623207046ae98a0a9368fff515b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45507 } }
I20260928 06:27:26.591991 18277 leader_election.cc:302] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26124623207046ae98a0a9368fff515b; no voters:
I20260928 06:27:26.592186 18277 leader_election.cc:288] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:26.592245 18282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:26.592406 18282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [term 1 LEADER]: Becoming Leader. State: Replica: 26124623207046ae98a0a9368fff515b, State: Running, Role: LEADER
I20260928 06:27:26.592461 18277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:26.592551 18282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26124623207046ae98a0a9368fff515b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45507 } }
I20260928 06:27:26.593000 18283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26124623207046ae98a0a9368fff515b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26124623207046ae98a0a9368fff515b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45507 } } }
I20260928 06:27:26.593076 18283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:26.593312 18284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26124623207046ae98a0a9368fff515b. Latest consensus state: current_term: 1 leader_uuid: "26124623207046ae98a0a9368fff515b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26124623207046ae98a0a9368fff515b" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45507 } } }
I20260928 06:27:26.593407 18284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:26.593642 18290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:26.594170 18290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:26.596028 18290 catalog_manager.cc:1347] Generated new cluster ID: 53017c52162047c2b34d68c393a284da
I20260928 06:27:26.596091 18290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:27:26 dist-test-slave-q03f krb5kdc[18196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:26.600740 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:26.588137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.31.214.126:45507)
0928 06:27:26.588290 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:26.588299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:26.588335 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:27:26.590255 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:26.590261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:26.590316 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:26.590442 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:26.591321 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.592477 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.593581 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.594206 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.594711 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:26.597340 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:26.597344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:26.597344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:26.597357 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:26.599118 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:26.599518 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:26.599521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:26.599521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:26.599660 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:26.599890 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:26.599890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:26.599891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:26.600139 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:26.600327 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:26.600394 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:27:26.600538 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260928 06:27:26.618592 18290 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:26.619369 18290 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:26.626241 18290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b: Generated new TSK 0
I20260928 06:27:26.626482 18290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:27:26.657596 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45507
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:26.744990 18303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:26.745256 18303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:26.745302 18303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:26.746647 18303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:26.746724 18303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:26.746771 18303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:27:26.748342 18303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:26.748636 18303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:26.748898 18303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:26.754359 18308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:26.754323 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:26.754478 18311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:26.754634 18303 server_base.cc:1034] running on GCE node
I20260928 06:27:26.754873 18303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:26.755160 18303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:26.756311 18303 hybrid_clock.cc:648] HybridClock initialized: now 1790576846756289 us; error 44 us; skew 500 ppm
Sep 28 06:27:26 dist-test-slave-q03f krb5kdc[18196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:26.759038 18303 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:27:26.759493 18303 webserver.cc:460] Webserver started at http://127.31.214.65:44473/ using document root <none> and password file <none>
I20260928 06:27:26.759742 18303 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:26.759809 18303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:26.759976 18303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:26.761148 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "386c6df8e90f4a638672b5ea0b5c3bdf"
format_stamp: "Formatted at 2026-09-28 06:27:26 on dist-test-slave-q03f"
server_key: "5d9907598e57d1c3218fad70b2390a0d"
server_key_iv: "34a5bfabd961d272656d9457c733e687"
server_key_version: "encryptionkey@0"
I20260928 06:27:26.761519 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "386c6df8e90f4a638672b5ea0b5c3bdf"
format_stamp: "Formatted at 2026-09-28 06:27:26 on dist-test-slave-q03f"
server_key: "5d9907598e57d1c3218fad70b2390a0d"
server_key_iv: "34a5bfabd961d272656d9457c733e687"
server_key_version: "encryptionkey@0"
I20260928 06:27:26.763172 18303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:27:26.764078 18318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:26.764309 18303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:27:26.764432 18303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "386c6df8e90f4a638672b5ea0b5c3bdf"
format_stamp: "Formatted at 2026-09-28 06:27:26 on dist-test-slave-q03f"
server_key: "5d9907598e57d1c3218fad70b2390a0d"
server_key_iv: "34a5bfabd961d272656d9457c733e687"
server_key_version: "encryptionkey@0"
I20260928 06:27:26.764510 18303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:26.775394 18303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:26.777926 18303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:26.778136 18303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:26.778407 18303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:26.778749 18303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:26.778806 18303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:26.778857 18303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:26.778959 18303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:26.785084 18303 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:38611
I20260928 06:27:26.785133 18431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:38611 every 8 connection(s)
I20260928 06:27:26.785480 18303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:27:26 dist-test-slave-q03f krb5kdc[18196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:26.794042 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 18303
I20260928 06:27:26.794186 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:27:26.794425 32601 external_mini_cluster.cc:1428] Setting key 77b32d73a47dfbe90ba5875a98132027
I20260928 06:27:26.796710 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:26.786391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:44273 (local address 127.31.214.126:45507)
0928 06:27:26.786478 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:26.786480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:26.786653 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:27:26.787251 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:26.787253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:26.787270 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:26.787330 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:26.788290 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.788850 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.789981 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.790267 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.790332 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:26.792924 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:26.792925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:26.792926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:26.792933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:26.794984 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:26.795566 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:26.795567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:26.795568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:26.795633 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:26.795941 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:26.795942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:26.795943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:26.796117 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:26.796189 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:26.796463 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 06:27:26.796572 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:27:26.797022 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:26.786317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45507 (local address 127.31.214.65:44273)
0928 06:27:26.786510 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:26.786550 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:26.787115 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:26.787385 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:26.787389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:26.787987 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:26.788198 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:26.788203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.788990 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.788991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:26.789844 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:26.789846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.790359 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.790360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:26.790479 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:26.791210 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:26.791220 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:26.792774 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:26.795068 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:26.795078 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:26.795081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:26.795484 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:26.795735 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:26.795736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:26.795736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:26.795862 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:26.796188 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:26.796195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:26.796327 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 06:27:26.796417 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:27:26.796565 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260928 06:27:26.797698 18432 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45507
I20260928 06:27:26.797813 18432 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:26.798069 18432 heartbeater.cc:511] Master 127.31.214.126:45507 requested a full tablet report, sending...
I20260928 06:27:26.798629 18241 ts_manager.cc:194] Registered new tserver with Master: 386c6df8e90f4a638672b5ea0b5c3bdf (127.31.214.65:38611)
I20260928 06:27:26.799449 18241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:44273
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:27:26.825856 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45507
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:26.915802 18437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:26.916038 18437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:26.916062 18437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:26.917177 18437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:26.917217 18437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:26.917243 18437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:27:26.918656 18437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:26.918845 18437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:26.919036 18437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:26.924325 18442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:26.924321 18443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:26.924360 18445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:26.924609 18437 server_base.cc:1034] running on GCE node
I20260928 06:27:26.925006 18437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:26.925228 18437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:26.926368 18437 hybrid_clock.cc:648] HybridClock initialized: now 1790576846926350 us; error 41 us; skew 500 ppm
Sep 28 06:27:26 dist-test-slave-q03f krb5kdc[18196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:26.928896 18437 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:27:26.929406 18437 webserver.cc:460] Webserver started at http://127.31.214.66:46751/ using document root <none> and password file <none>
I20260928 06:27:26.929600 18437 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:26.929642 18437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:26.929734 18437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:26.930862 18437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "f22fc650ff894d41a8fd6f0e9684e0df"
format_stamp: "Formatted at 2026-09-28 06:27:26 on dist-test-slave-q03f"
server_key: "9b23e2c6c7f8ba446774c93ce5a65c47"
server_key_iv: "6f2700f60d354abbeaa0634cc4b96fa8"
server_key_version: "encryptionkey@0"
I20260928 06:27:26.931277 18437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "f22fc650ff894d41a8fd6f0e9684e0df"
format_stamp: "Formatted at 2026-09-28 06:27:26 on dist-test-slave-q03f"
server_key: "9b23e2c6c7f8ba446774c93ce5a65c47"
server_key_iv: "6f2700f60d354abbeaa0634cc4b96fa8"
server_key_version: "encryptionkey@0"
I20260928 06:27:26.932711 18437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:27:26.933629 18452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:26.933830 18437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:26.934047 18437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "f22fc650ff894d41a8fd6f0e9684e0df"
format_stamp: "Formatted at 2026-09-28 06:27:26 on dist-test-slave-q03f"
server_key: "9b23e2c6c7f8ba446774c93ce5a65c47"
server_key_iv: "6f2700f60d354abbeaa0634cc4b96fa8"
server_key_version: "encryptionkey@0"
I20260928 06:27:26.934137 18437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:26.943425 18437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:26.946609 18437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:26.946789 18437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:26.947037 18437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:26.947381 18437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:26.947433 18437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:26.947503 18437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:26.947535 18437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:26.954910 18437 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:43461
I20260928 06:27:26.954978 18565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:43461 every 8 connection(s)
I20260928 06:27:26.955330 18437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:27:26 dist-test-slave-q03f krb5kdc[18196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:26.962042 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 18437
I20260928 06:27:26.962170 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:27:26.962374 32601 external_mini_cluster.cc:1428] Setting key b109c8ecedd2906e4d5ee316cf8c766d
I20260928 06:27:26.965708 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:26.956118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:53311 (local address 127.31.214.126:45507)
0928 06:27:26.956186 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:26.956188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:26.956403 (+ 215us) server_negotiation.cc:407] Connection header received
0928 06:27:26.956909 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:26.956910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:26.956922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:26.957003 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:26.958130 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.958669 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.959765 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.959968 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.960026 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:26.962452 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:26.962454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:26.962454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:26.962463 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:26.964096 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:26.964626 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:26.964627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:26.964627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:26.964680 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:26.964994 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:26.964995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:26.964996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:26.965157 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:26.965233 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:26.965486 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 06:27:26.965589 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:27:26.965898 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:26.956078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45507 (local address 127.31.214.66:53311)
0928 06:27:26.956249 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:26.956292 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:26.956785 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:26.957113 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:26.957117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:26.957733 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:26.958024 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:26.958032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.958764 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.958765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:26.959677 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:26.959680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:26.960057 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:26.960058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:26.960168 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:26.960756 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:26.960765 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:26.962355 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:26.964186 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:26.964194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:26.964196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:26.964549 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:26.964745 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:26.964746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:26.964746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:26.964910 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:26.965227 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:26.965232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:26.965354 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:27:26.965440 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:27:26.965599 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260928 06:27:26.966473 18566 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45507
I20260928 06:27:26.966584 18566 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:26.966786 18566 heartbeater.cc:511] Master 127.31.214.126:45507 requested a full tablet report, sending...
I20260928 06:27:26.967231 18241 ts_manager.cc:194] Registered new tserver with Master: f22fc650ff894d41a8fd6f0e9684e0df (127.31.214.66:43461)
I20260928 06:27:26.967643 18241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:53311
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:27:26.989804 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45507
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:41731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:27.076000 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:27.076207 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:27.076231 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:27.077379 18571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:27.077427 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:27.077454 18571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:27:27.079008 18571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:27.079257 18571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:27.079507 18571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:27.084820 18576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:27.084813 18579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:27.084813 18577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:27.084880 18571 server_base.cc:1034] running on GCE node
I20260928 06:27:27.085222 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:27.085431 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:27.086583 18571 hybrid_clock.cc:648] HybridClock initialized: now 1790576847086555 us; error 43 us; skew 500 ppm
Sep 28 06:27:27 dist-test-slave-q03f krb5kdc[18196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576847, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:27.089556 18571 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:27:27.090137 18571 webserver.cc:460] Webserver started at http://127.31.214.67:45393/ using document root <none> and password file <none>
I20260928 06:27:27.090394 18571 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:27.090478 18571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:27.090641 18571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:27.091851 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "b30b0c7c602b4d2d88ecc4266921379d"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "175401cf2a39c1134cf00d6f1abff1e8"
server_key_iv: "3cd4cbd3ede5906532d2086107c41574"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.092384 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "b30b0c7c602b4d2d88ecc4266921379d"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "175401cf2a39c1134cf00d6f1abff1e8"
server_key_iv: "3cd4cbd3ede5906532d2086107c41574"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.094097 18571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:27:27.095117 18586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.095333 18571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:27.095445 18571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "b30b0c7c602b4d2d88ecc4266921379d"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "175401cf2a39c1134cf00d6f1abff1e8"
server_key_iv: "3cd4cbd3ede5906532d2086107c41574"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.095566 18571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:27.106595 18571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:27.108731 18571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:27.108901 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:27.109169 18571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:27.109511 18571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:27.109566 18571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.109612 18571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:27.109647 18571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.115811 18571 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:39013
I20260928 06:27:27.115903 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:39013 every 8 connection(s)
I20260928 06:27:27.116236 18571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:27:27 dist-test-slave-q03f krb5kdc[18196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576847, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:27.125870 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.116995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:54837 (local address 127.31.214.126:45507)
0928 06:27:27.117109 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.117112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.117287 (+ 175us) server_negotiation.cc:407] Connection header received
0928 06:27:27.117773 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.117774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.117784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.117838 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:27.118847 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.119390 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.120397 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.120585 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.120643 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.122988 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:27.122989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:27.122989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:27.122998 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:27.124498 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:27.124916 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:27.124918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:27.124918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:27.124981 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:27.125274 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:27.125275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:27.125275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:27.125402 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:27.125464 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.125723 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.125802 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 06:27:27.125965 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 18571
I20260928 06:27:27.126063 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:27:27.126277 32601 external_mini_cluster.cc:1428] Setting key 3d7e2be50013eb3966da27453095dbc2
I20260928 06:27:27.126371 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.116980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45507 (local address 127.31.214.67:54837)
0928 06:27:27.117151 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.117192 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:27.117681 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:27.117895 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:27.117899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:27.118555 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:27.118755 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.118760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.119482 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.119483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.120313 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.120315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.120671 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.120673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.120786 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.121385 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:27.121393 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:27.122871 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:27.124588 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:27.124596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:27.124598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:27.124829 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:27.125083 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:27.125084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:27.125084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:27.125190 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:27.125466 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:27.125470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:27.125605 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:27:27.125691 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:27:27.125940 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260928 06:27:27.127017 18700 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45507
I20260928 06:27:27.127122 18700 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:27.127327 18700 heartbeater.cc:511] Master 127.31.214.126:45507 requested a full tablet report, sending...
I20260928 06:27:27.127708 18241 ts_manager.cc:194] Registered new tserver with Master: b30b0c7c602b4d2d88ecc4266921379d (127.31.214.67:39013)
I20260928 06:27:27.128115 18241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:54837
I20260928 06:27:27.138298 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:27:27.146881 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.139872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.31.214.126:45507)
0928 06:27:27.139925 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.139927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.139991 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:27:27.140062 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.140063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.140072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.140119 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:27.140726 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.141270 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.142105 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.142292 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.142347 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.143791 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:27.143793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:27.143793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:27.143800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:27.145352 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:27.145828 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:27.145829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:27.145830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:27.145907 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:27.146258 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:27.146259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:27.146259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:27.146418 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:27.146505 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.146645 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.146778 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:27:27.148844 18241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:27:27.151054 18241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:27:27.159071 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.154095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:43461 (local address 127.0.0.1:39082)
0928 06:27:27.154669 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.154680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:27.154777 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:27.155044 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:27.155046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:27.155049 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:27.155169 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.155171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.156210 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.156211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.157701 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.157704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.158653 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.158654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.158748 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.158763 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:27:27.158909 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 06:27:27.158974 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":28,"thread_start_us":79,"threads_started":1}
I20260928 06:27:27.159325 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.154237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.31.214.66:43461)
0928 06:27:27.154482 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.154488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.154745 (+ 257us) server_negotiation.cc:407] Connection header received
0928 06:27:27.154872 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.154876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.154893 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.154972 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:27.155264 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.156086 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.157820 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.158563 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.158647 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.158713 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.159195 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.159248 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
I20260928 06:27:27.159996 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.154314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:39013 (local address 127.0.0.1:35260)
0928 06:27:27.154766 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.154776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:27.154832 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:27.156240 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:27.156241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:27.156244 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:27.156284 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.156287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.157343 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.157344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.158380 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.158383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.159734 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.159735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.159803 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.159807 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:27.159872 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:27:27.159918 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":77,"threads_started":1}
I20260928 06:27:27.160065 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.154427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.31.214.67:39013)
0928 06:27:27.154773 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.154778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.155085 (+ 307us) server_negotiation.cc:407] Connection header received
0928 06:27:27.155131 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.155133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.155151 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.155231 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:27.156364 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.157234 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.158482 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.159349 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.159733 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.159805 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.159887 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.159935 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":129,"threads_started":1}
I20260928 06:27:27.160130 18500 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6858c932014abd9074e66caf6ff97a (DEFAULT_TABLE table=test-table [id=5e636a909c1f4826b357bdf856972f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:27.160501 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.154094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:38611 (local address 127.0.0.1:33046)
0928 06:27:27.154474 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.154526 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:27.154633 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:27.155042 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:27.155046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:27.155052 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:27.155169 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.155172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.157765 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.157766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.158814 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.158816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.160240 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.160242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.160306 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.160310 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:27.160366 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:27:27.160407 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260928 06:27:27.160532 18500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6858c932014abd9074e66caf6ff97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:27.160756 18713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.154237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.31.214.65:38611)
0928 06:27:27.154583 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.154587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.154615 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:27:27.154908 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.154911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.154928 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.155020 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:27.156385 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.157645 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.158901 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.159924 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.159995 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.160047 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.160627 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.160675 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":102,"threads_started":1}
I20260928 06:27:27.161005 18634 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6858c932014abd9074e66caf6ff97a (DEFAULT_TABLE table=test-table [id=5e636a909c1f4826b357bdf856972f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:27.161362 18634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6858c932014abd9074e66caf6ff97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:27.161449 18366 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6858c932014abd9074e66caf6ff97a (DEFAULT_TABLE table=test-table [id=5e636a909c1f4826b357bdf856972f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:27.161772 18366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6858c932014abd9074e66caf6ff97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:27.163302 18717 tablet_bootstrap.cc:492] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: Bootstrap starting.
I20260928 06:27:27.164013 18717 tablet_bootstrap.cc:654] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:27.164448 18717 log.cc:824] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:27.165122 18719 tablet_bootstrap.cc:492] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: Bootstrap starting.
I20260928 06:27:27.165191 18717 tablet_bootstrap.cc:492] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: No bootstrap required, opened a new log
I20260928 06:27:27.165247 18717 ts_tablet_manager.cc:1397] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:27:27.165828 18719 tablet_bootstrap.cc:654] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:27.165933 18720 tablet_bootstrap.cc:492] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf: Bootstrap starting.
I20260928 06:27:27.166471 18719 log.cc:824] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:27.166749 18720 tablet_bootstrap.cc:654] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:27.167445 18719 tablet_bootstrap.cc:492] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: No bootstrap required, opened a new log
I20260928 06:27:27.167522 18719 ts_tablet_manager.cc:1397] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:27:27.167534 18720 log.cc:824] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:27.167896 18717 raft_consensus.cc:348] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } }
I20260928 06:27:27.168085 18717 raft_consensus.cc:374] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:27.168126 18717 raft_consensus.cc:729] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22fc650ff894d41a8fd6f0e9684e0df, State: Initialized, Role: FOLLOWER
I20260928 06:27:27.168241 18717 consensus_queue.cc:260] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } }
I20260928 06:27:27.168375 18720 tablet_bootstrap.cc:492] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf: No bootstrap required, opened a new log
I20260928 06:27:27.168447 18720 ts_tablet_manager.cc:1397] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:27:27.168546 18717 ts_tablet_manager.cc:1428] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:27:27.168692 18566 heartbeater.cc:503] Master 127.31.214.126:45507 was elected leader, sending a full tablet report...
I20260928 06:27:27.170156 18719 raft_consensus.cc:348] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } }
I20260928 06:27:27.170367 18719 raft_consensus.cc:374] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:27.170413 18719 raft_consensus.cc:729] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30b0c7c602b4d2d88ecc4266921379d, State: Initialized, Role: FOLLOWER
I20260928 06:27:27.170535 18719 consensus_queue.cc:260] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } }
I20260928 06:27:27.170836 18719 ts_tablet_manager.cc:1428] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:27:27.170873 18700 heartbeater.cc:503] Master 127.31.214.126:45507 was elected leader, sending a full tablet report...
I20260928 06:27:27.170792 18720 raft_consensus.cc:348] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } }
I20260928 06:27:27.171010 18720 raft_consensus.cc:374] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:27.171058 18720 raft_consensus.cc:729] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386c6df8e90f4a638672b5ea0b5c3bdf, State: Initialized, Role: FOLLOWER
I20260928 06:27:27.171177 18720 consensus_queue.cc:260] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } }
I20260928 06:27:27.171494 18720 ts_tablet_manager.cc:1428] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:27:27.171541 18432 heartbeater.cc:503] Master 127.31.214.126:45507 was elected leader, sending a full tablet report...
W20260928 06:27:27.205920 18567 tablet.cc:2369] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:27:27.276502 18724 raft_consensus.cc:482] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:27.276640 18724 raft_consensus.cc:504] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } }
I20260928 06:27:27.277243 18724 leader_election.cc:288] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f22fc650ff894d41a8fd6f0e9684e0df (127.31.214.66:43461), 386c6df8e90f4a638672b5ea0b5c3bdf (127.31.214.65:38611)
I20260928 06:27:27.280592 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.277440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:38611 (local address 127.31.214.67:38455)
0928 06:27:27.277498 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.277510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:27.277558 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:27.277780 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:27.277781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:27.277783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:27.277861 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.277863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.278791 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.278792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.279672 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.279674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.280294 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.280295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.280384 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.280387 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:27.280431 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:27:27.280495 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":129}
I20260928 06:27:27.280598 18713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.277503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:38455 (local address 127.31.214.65:38611)
0928 06:27:27.277546 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.277548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.277560 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:27:27.277705 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.277706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.277718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.277775 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:27.278053 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.278669 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.279778 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.280204 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.280289 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.280300 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.280467 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.280516 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:27:27.280617 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.277644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:43975 (local address 127.31.214.66:43461)
0928 06:27:27.277712 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.277714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.277840 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:27:27.277895 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.277896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.277907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.277987 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:27.278156 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.278660 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.279818 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.280211 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.280303 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.280312 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.280507 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.280555 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:27:27.280620 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.277565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:43461 (local address 127.31.214.67:43975)
0928 06:27:27.277770 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.277785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:27.277848 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:27.277996 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:27.277997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:27.277999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:27.278082 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.278084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.278748 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.278750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.279691 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.279693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.280320 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.280322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.280392 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.280395 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:27.280485 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:27:27.280525 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1,"thread_start_us":147,"threads_started":1}
I20260928 06:27:27.281009 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6858c932014abd9074e66caf6ff97a" candidate_uuid: "b30b0c7c602b4d2d88ecc4266921379d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" is_pre_election: true
I20260928 06:27:27.281222 18386 raft_consensus.cc:2454] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b30b0c7c602b4d2d88ecc4266921379d in term 0.
I20260928 06:27:27.281248 18519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6858c932014abd9074e66caf6ff97a" candidate_uuid: "b30b0c7c602b4d2d88ecc4266921379d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" is_pre_election: true
I20260928 06:27:27.281376 18519 raft_consensus.cc:2454] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b30b0c7c602b4d2d88ecc4266921379d in term 0.
I20260928 06:27:27.281543 18589 leader_election.cc:302] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386c6df8e90f4a638672b5ea0b5c3bdf, b30b0c7c602b4d2d88ecc4266921379d; no voters:
I20260928 06:27:27.281714 18724 raft_consensus.cc:2781] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:27:27.281797 18724 raft_consensus.cc:482] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:27:27.281831 18724 raft_consensus.cc:3037] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:27.282624 18724 raft_consensus.cc:504] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } }
I20260928 06:27:27.282778 18724 leader_election.cc:288] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [CANDIDATE]: Term 1 election: Requested vote from peers f22fc650ff894d41a8fd6f0e9684e0df (127.31.214.66:43461), 386c6df8e90f4a638672b5ea0b5c3bdf (127.31.214.65:38611)
I20260928 06:27:27.282927 18519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6858c932014abd9074e66caf6ff97a" candidate_uuid: "b30b0c7c602b4d2d88ecc4266921379d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f22fc650ff894d41a8fd6f0e9684e0df"
I20260928 06:27:27.282979 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6858c932014abd9074e66caf6ff97a" candidate_uuid: "b30b0c7c602b4d2d88ecc4266921379d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf"
I20260928 06:27:27.283032 18519 raft_consensus.cc:3037] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:27.283079 18386 raft_consensus.cc:3037] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:27.283815 18519 raft_consensus.cc:2454] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30b0c7c602b4d2d88ecc4266921379d in term 1.
I20260928 06:27:27.283818 18386 raft_consensus.cc:2454] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30b0c7c602b4d2d88ecc4266921379d in term 1.
I20260928 06:27:27.284013 18587 leader_election.cc:302] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b30b0c7c602b4d2d88ecc4266921379d, f22fc650ff894d41a8fd6f0e9684e0df; no voters:
I20260928 06:27:27.284134 18724 raft_consensus.cc:2781] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:27.284302 18724 raft_consensus.cc:686] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 1 LEADER]: Becoming Leader. State: Replica: b30b0c7c602b4d2d88ecc4266921379d, State: Running, Role: LEADER
I20260928 06:27:27.284416 18724 consensus_queue.cc:237] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } }
I20260928 06:27:27.285455 18241 catalog_manager.cc:5495] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d reported cstate change: term changed from 0 to 1, leader changed from <none> to b30b0c7c602b4d2d88ecc4266921379d (127.31.214.67). New cstate: current_term: 1 leader_uuid: "b30b0c7c602b4d2d88ecc4266921379d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b30b0c7c602b4d2d88ecc4266921379d" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 } health_report { overall_health: UNKNOWN } } }
W20260928 06:27:27.286319 18433 tablet.cc:2369] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:27:27.305289 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.302560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.31.214.67:39013)
0928 06:27:27.302640 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.302643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.302652 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:27:27.302702 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.302703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.302722 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.302811 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:27.303029 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.303567 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.304416 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.304638 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.304720 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.304883 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:27.305000 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:27.305127 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.305161 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.305175 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:27:27.307729 18386 raft_consensus.cc:1261] T 6e6858c932014abd9074e66caf6ff97a P 386c6df8e90f4a638672b5ea0b5c3bdf [term 1 FOLLOWER]: Refusing update from remote peer b30b0c7c602b4d2d88ecc4266921379d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:27.307729 18519 raft_consensus.cc:1261] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [term 1 FOLLOWER]: Refusing update from remote peer b30b0c7c602b4d2d88ecc4266921379d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:27.308173 18727 consensus_queue.cc:1035] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f22fc650ff894d41a8fd6f0e9684e0df" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:27.308319 18724 consensus_queue.cc:1035] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [LEADER]: Connected to new peer: Peer: permanent_uuid: "386c6df8e90f4a638672b5ea0b5c3bdf" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:27.311089 18730 mvcc.cc:205] Tried to move back new op lower bound from 7334202766568730624 to 7334202766477852672. Current Snapshot: MvccSnapshot[applied={T|T < 7334202766568730624}]
I20260928 06:27:27.311373 18731 mvcc.cc:205] Tried to move back new op lower bound from 7334202766568730624 to 7334202766477852672. Current Snapshot: MvccSnapshot[applied={T|T < 7334202766568730624}]
I20260928 06:27:27.311990 18732 mvcc.cc:205] Tried to move back new op lower bound from 7334202766568730624 to 7334202766477852672. Current Snapshot: MvccSnapshot[applied={T|T < 7334202766568730624}]
I20260928 06:27:27.314168 18241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:27.314277 18241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:27.315083 18241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26124623207046ae98a0a9368fff515b: Sending DeleteTablet for 3 replicas of tablet 6e6858c932014abd9074e66caf6ff97a
I20260928 06:27:27.315476 18366 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6858c932014abd9074e66caf6ff97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:27 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:33046
I20260928 06:27:27.315511 18500 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6858c932014abd9074e66caf6ff97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:27 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:39082
I20260928 06:27:27.315594 18634 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6858c932014abd9074e66caf6ff97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:27 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:35260
I20260928 06:27:27.315827 18741 tablet_replica.cc:330] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: stopping tablet replica
I20260928 06:27:27.315924 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 18303
I20260928 06:27:27.315944 18741 raft_consensus.cc:2229] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:27:27.316488 18741 raft_consensus.cc:2258] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:27.316792 18740 tablet_replica.cc:330] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: stopping tablet replica
I20260928 06:27:27.316921 18740 raft_consensus.cc:2229] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:27.317058 18740 raft_consensus.cc:2258] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:27.317410 18741 ts_tablet_manager.cc:1905] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:27.320015 18740 ts_tablet_manager.cc:1905] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:27.322101 18741 ts_tablet_manager.cc:1918] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:27.322157 18740 ts_tablet_manager.cc:1918] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:27.322240 18740 log.cc:1196] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/6e6858c932014abd9074e66caf6ff97a
I20260928 06:27:27.322333 18741 log.cc:1196] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/6e6858c932014abd9074e66caf6ff97a
I20260928 06:27:27.322551 18740 ts_tablet_manager.cc:1939] T 6e6858c932014abd9074e66caf6ff97a P f22fc650ff894d41a8fd6f0e9684e0df: Deleting consensus metadata
I20260928 06:27:27.322764 18741 ts_tablet_manager.cc:1939] T 6e6858c932014abd9074e66caf6ff97a P b30b0c7c602b4d2d88ecc4266921379d: Deleting consensus metadata
I20260928 06:27:27.323043 18225 catalog_manager.cc:4841] TS f22fc650ff894d41a8fd6f0e9684e0df (127.31.214.66:43461): tablet 6e6858c932014abd9074e66caf6ff97a (table test-table [id=5e636a909c1f4826b357bdf856972f68]) successfully deleted
I20260928 06:27:27.323182 18225 catalog_manager.cc:4841] TS b30b0c7c602b4d2d88ecc4266921379d (127.31.214.67:39013): tablet 6e6858c932014abd9074e66caf6ff97a (table test-table [id=5e636a909c1f4826b357bdf856972f68]) successfully deleted
W20260928 06:27:27.325048 18227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:38611 (error 108)
I20260928 06:27:27.325191 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 18437
I20260928 06:27:27.327879 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.327667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:38611 (local address 127.0.0.1:33050)
0928 06:27:27.327749 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.327804 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 06:27:27.328033 18227 catalog_manager.cc:4584] TS 386c6df8e90f4a638672b5ea0b5c3bdf (127.31.214.65:38611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e6858c932014abd9074e66caf6ff97a: Network error: Client connection negotiation failed: client connection to 127.31.214.65:38611: connect: Connection refused (error 111)
I20260928 06:27:27.331162 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 18571
I20260928 06:27:27.337107 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 18209
2026-09-28T06:27:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2053 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T06:27:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:27Z Disabled control of system clock
I20260928 06:27:27.365564 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45225
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:45225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:27:27.449735 18746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:27.450042 18746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:27.450075 18746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:27.451347 18746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:27.451406 18746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:27.451418 18746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:27.451431 18746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:27.451443 18746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:27.452917 18746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:45225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:45225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:27.453265 18746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:27.453617 18746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:27.458930 18751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:27.458905 18754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:27.458905 18752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:27.459442 18746 server_base.cc:1034] running on GCE node
I20260928 06:27:27.459740 18746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:27.460114 18746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:27.461308 18746 hybrid_clock.cc:648] HybridClock initialized: now 1790576847461288 us; error 49 us; skew 500 ppm
I20260928 06:27:27.468226 18746 webserver.cc:460] Webserver started at http://127.31.214.126:42761/ using document root <none> and password file <none>
I20260928 06:27:27.468541 18746 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:27.468626 18746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:27.468784 18746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:27.470069 18746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "e51b3de4bc8e4836a7285e394eedd9c2"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "bec4642194519aba7576d87d3b961ba3"
server_key_iv: "5605be187f2fe10e332cc2adc67316a9"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.470465 18746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "e51b3de4bc8e4836a7285e394eedd9c2"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "bec4642194519aba7576d87d3b961ba3"
server_key_iv: "5605be187f2fe10e332cc2adc67316a9"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.471849 18746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:27:27.472700 18760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.472886 18746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:27.472996 18746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "e51b3de4bc8e4836a7285e394eedd9c2"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "bec4642194519aba7576d87d3b961ba3"
server_key_iv: "5605be187f2fe10e332cc2adc67316a9"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.473083 18746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:27.503096 18746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:27.503453 18746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:27.503625 18746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:27.508334 18746 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:45225
I20260928 06:27:27.508371 18812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:45225 every 8 connection(s)
I20260928 06:27:27.508781 18746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:27.509876 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:27.510727 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 18746
I20260928 06:27:27.510849 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:27.511060 32601 external_mini_cluster.cc:1428] Setting key 94ee4e0bbe7bb0905f5cf25711bc3189
I20260928 06:27:27.513149 18813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2: Bootstrap starting.
I20260928 06:27:27.513906 18813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:27.514271 18813 log.cc:824] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:27.515069 18813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2: No bootstrap required, opened a new log
I20260928 06:27:27.516829 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.511630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.31.214.126:45225)
0928 06:27:27.511826 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.511834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.511859 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:27:27.512276 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.512284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.512349 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.512467 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:27.512758 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.513670 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.514861 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.515305 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.515397 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.516144 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:27.516148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:27.516148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:27.516309 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:27.516340 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:27.516350 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:27.516401 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.516524 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.516590 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260928 06:27:27.517601 18813 raft_consensus.cc:348] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51b3de4bc8e4836a7285e394eedd9c2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45225 } }
I20260928 06:27:27.517764 18813 raft_consensus.cc:374] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:27.517832 18813 raft_consensus.cc:729] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51b3de4bc8e4836a7285e394eedd9c2, State: Initialized, Role: FOLLOWER
I20260928 06:27:27.518016 18813 consensus_queue.cc:260] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51b3de4bc8e4836a7285e394eedd9c2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45225 } }
I20260928 06:27:27.518108 18813 raft_consensus.cc:388] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:27.518152 18813 raft_consensus.cc:482] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:27.518204 18813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:27.518785 18813 raft_consensus.cc:504] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51b3de4bc8e4836a7285e394eedd9c2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45225 } }
I20260928 06:27:27.518906 18813 leader_election.cc:302] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e51b3de4bc8e4836a7285e394eedd9c2; no voters:
I20260928 06:27:27.519122 18813 leader_election.cc:288] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:27.519182 18818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:27.519367 18818 raft_consensus.cc:686] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [term 1 LEADER]: Becoming Leader. State: Replica: e51b3de4bc8e4836a7285e394eedd9c2, State: Running, Role: LEADER
I20260928 06:27:27.519423 18813 sys_catalog.cc:564] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:27.519459 18818 consensus_queue.cc:237] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51b3de4bc8e4836a7285e394eedd9c2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45225 } }
I20260928 06:27:27.519865 18819 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e51b3de4bc8e4836a7285e394eedd9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51b3de4bc8e4836a7285e394eedd9c2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45225 } } }
I20260928 06:27:27.519881 18820 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e51b3de4bc8e4836a7285e394eedd9c2. Latest consensus state: current_term: 1 leader_uuid: "e51b3de4bc8e4836a7285e394eedd9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e51b3de4bc8e4836a7285e394eedd9c2" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 45225 } } }
I20260928 06:27:27.519996 18819 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:27.520005 18820 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:27.520241 18825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:27.520790 18825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:27.522548 18825 catalog_manager.cc:1347] Generated new cluster ID: d42b0f8615694c6981762980158057a5
I20260928 06:27:27.522627 18825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:27.530511 18825 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:27.531312 18825 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:27.536530 18825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2: Generated new TSK 0
I20260928 06:27:27.536759 18825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:27.541138 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45225
--builtin_ntp_servers=127.31.214.84:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:27:27.632607 18837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:27.632866 18837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:27.632907 18837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:27.634286 18837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:27.634358 18837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:27.634405 18837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:27:27.635921 18837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:27.636222 18837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:27.636624 18837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:27.643697 18843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:27.643981 18842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:27.643712 18845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:27.645038 18837 server_base.cc:1034] running on GCE node
I20260928 06:27:27.645371 18837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:27.645831 18837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:27.647126 18837 hybrid_clock.cc:648] HybridClock initialized: now 1790576847647112 us; error 75 us; skew 500 ppm
I20260928 06:27:27.649348 18837 webserver.cc:460] Webserver started at http://127.31.214.65:46179/ using document root <none> and password file <none>
I20260928 06:27:27.649747 18837 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:27.649827 18837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:27.650102 18837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:27.651520 18837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "72389f92a63345fdb69235cca8b90d4d"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "7fd61d680406849897ba9cd986b20269"
server_key_iv: "387264f1a47047570ba545aadf899a74"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.651957 18837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "72389f92a63345fdb69235cca8b90d4d"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "7fd61d680406849897ba9cd986b20269"
server_key_iv: "387264f1a47047570ba545aadf899a74"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.656317 18837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 06:27:27.660025 18851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.660987 18837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:27:27.661382 18837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "72389f92a63345fdb69235cca8b90d4d"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "7fd61d680406849897ba9cd986b20269"
server_key_iv: "387264f1a47047570ba545aadf899a74"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.661795 18837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:27.703002 18837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:27.703382 18837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:27.703562 18837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:27.703886 18837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:27.704332 18837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:27.704368 18837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.704432 18837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:27.704456 18837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.711557 18837 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:46043
I20260928 06:27:27.711604 18964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:46043 every 8 connection(s)
I20260928 06:27:27.712015 18837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:27:27.718477 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.712935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:54949 (local address 127.31.214.126:45225)
0928 06:27:27.713002 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.713004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.713277 (+ 273us) server_negotiation.cc:407] Connection header received
0928 06:27:27.713876 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.713877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.713896 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.713989 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:27.714745 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.715490 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.716617 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.716831 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.716973 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.718132 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:27.718134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:27.718134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:27.718140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:27.718156 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:27.718161 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:27.718234 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.718339 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.718376 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:27:27.718740 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.712932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45225 (local address 127.31.214.65:54949)
0928 06:27:27.713146 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.713184 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:27.713741 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:27.714057 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:27.714066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:27.714075 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:27.714638 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.714644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.715628 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.715629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.716515 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.716519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.716992 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.716993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.717262 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.717885 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:27:27.717885 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:27.717924 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:27:27.718016 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:27.718235 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:27.718249 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:27.718252 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:27.718344 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:27:27.718460 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260928 06:27:27.718852 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 18837
I20260928 06:27:27.718940 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:27:27.719163 32601 external_mini_cluster.cc:1428] Setting key 55fc37422e2caeb2bd90b6f3ac982843
I20260928 06:27:27.719352 18965 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45225
I20260928 06:27:27.719475 18965 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:27.719688 18965 heartbeater.cc:511] Master 127.31.214.126:45225 requested a full tablet report, sending...
I20260928 06:27:27.720289 18777 ts_manager.cc:194] Registered new tserver with Master: 72389f92a63345fdb69235cca8b90d4d (127.31.214.65:46043)
I20260928 06:27:27.720392 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45225
--builtin_ntp_servers=127.31.214.84:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:27:27.721272 18777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.65:54949
W20260928 06:27:27.806622 18968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:27.806840 18968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:27.806862 18968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:27.808072 18968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:27.808121 18968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:27.808149 18968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:27:27.809558 18968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:27.809779 18968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:27.810081 18968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:27.815974 18973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:27.816004 18974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:27.816135 18968 server_base.cc:1034] running on GCE node
W20260928 06:27:27.815966 18976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:27.816581 18968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:27.816905 18968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:27.818090 18968 hybrid_clock.cc:648] HybridClock initialized: now 1790576847818077 us; error 39 us; skew 500 ppm
I20260928 06:27:27.819588 18968 webserver.cc:460] Webserver started at http://127.31.214.66:43157/ using document root <none> and password file <none>
I20260928 06:27:27.819864 18968 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:27.819957 18968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:27.820132 18968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:27.821420 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "97d54174252a442aaacbbe155dfc77e9"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "3b0e40b51934736836972360aae0466a"
server_key_iv: "df2b13ad6520813208e8344e91470284"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.821811 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "97d54174252a442aaacbbe155dfc77e9"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "3b0e40b51934736836972360aae0466a"
server_key_iv: "df2b13ad6520813208e8344e91470284"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.823295 18968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:27.824211 18982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.824419 18968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:27.824532 18968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "97d54174252a442aaacbbe155dfc77e9"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "3b0e40b51934736836972360aae0466a"
server_key_iv: "df2b13ad6520813208e8344e91470284"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.824609 18968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:27.844081 18968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:27.844508 18968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:27.844692 18968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:27.844995 18968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:27.845404 18968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:27.845465 18968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.845513 18968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:27.845549 18968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.852309 18968 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:43537
I20260928 06:27:27.852394 19095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:43537 every 8 connection(s)
I20260928 06:27:27.852788 18968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:27:27.856106 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 18968
I20260928 06:27:27.856205 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:27:27.856424 32601 external_mini_cluster.cc:1428] Setting key 11246a9f331e59421cbd094a80ca6c40
I20260928 06:27:27.857583 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45225
--builtin_ntp_servers=127.31.214.84:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:27:27.860339 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.853696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:55463 (local address 127.31.214.126:45225)
0928 06:27:27.853793 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.853796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.854219 (+ 423us) server_negotiation.cc:407] Connection header received
0928 06:27:27.854777 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.854778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.854789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.854851 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:27.855952 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.856755 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.858468 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.858771 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.858846 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.860014 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:27.860015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:27.860016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:27.860024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:27.860043 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:27.860048 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:27.860113 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.860242 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.860278 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:27:27.860589 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.853673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45225 (local address 127.31.214.66:55463)
0928 06:27:27.854060 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.854104 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:27.854652 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:27.855013 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:27.855016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:27.855025 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:27.855799 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.855807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.856884 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.856885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.858362 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.858366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.858873 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.858874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.859064 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.859780 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:27:27.859780 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:27.859817 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:27:27.859847 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:27.860141 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:27.860148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:27.860151 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:27.860226 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:27:27.860323 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
I20260928 06:27:27.861084 19096 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45225
I20260928 06:27:27.861210 19096 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:27.861406 19096 heartbeater.cc:511] Master 127.31.214.126:45225 requested a full tablet report, sending...
I20260928 06:27:27.861836 18777 ts_manager.cc:194] Registered new tserver with Master: 97d54174252a442aaacbbe155dfc77e9 (127.31.214.66:43537)
I20260928 06:27:27.862260 18777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.66:55463
W20260928 06:27:27.942584 19099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:27.942843 19099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:27.942875 19099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:27.944023 19099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:27.944101 19099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:27.944144 19099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:27:27.945526 19099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:27.945802 19099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:27.946187 19099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:27.951565 19105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:27.951599 19104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:27.951706 19107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:27.951813 19099 server_base.cc:1034] running on GCE node
I20260928 06:27:27.952122 19099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:27.952387 19099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:27.953560 19099 hybrid_clock.cc:648] HybridClock initialized: now 1790576847953541 us; error 39 us; skew 500 ppm
I20260928 06:27:27.954988 19099 webserver.cc:460] Webserver started at http://127.31.214.67:33169/ using document root <none> and password file <none>
I20260928 06:27:27.955255 19099 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:27.955329 19099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:27.955518 19099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:27.956655 19099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "e39215a3b9e644c790b4959f5692ef20"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "4abf0b4c5099ff338764a9098a8cb399"
server_key_iv: "c0471012c3d1bf79bbdeef6008e95b1b"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.957027 19099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "e39215a3b9e644c790b4959f5692ef20"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "4abf0b4c5099ff338764a9098a8cb399"
server_key_iv: "c0471012c3d1bf79bbdeef6008e95b1b"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.958482 19099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:27:27.959338 19113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.959555 19099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:27.959666 19099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "e39215a3b9e644c790b4959f5692ef20"
format_stamp: "Formatted at 2026-09-28 06:27:27 on dist-test-slave-q03f"
server_key: "4abf0b4c5099ff338764a9098a8cb399"
server_key_iv: "c0471012c3d1bf79bbdeef6008e95b1b"
server_key_version: "encryptionkey@0"
I20260928 06:27:27.959743 19099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:27.977716 19099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:27.978132 19099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:27.978302 19099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:27.978569 19099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:27.978919 19099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:27.978973 19099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.979013 19099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:27.979049 19099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:27.985562 19099 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:34833
I20260928 06:27:27.985610 19226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:34833 every 8 connection(s)
I20260928 06:27:27.986061 19099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:27:27.991647 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.986922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:49797 (local address 127.31.214.126:45225)
0928 06:27:27.986979 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.986982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.987229 (+ 247us) server_negotiation.cc:407] Connection header received
0928 06:27:27.987716 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.987717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.987729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.987805 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:27.988487 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.989063 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.990195 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.990402 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.990480 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.991321 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:27.991323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:27.991323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:27.991328 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:27.991367 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:27.991371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:27.991432 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:27.991550 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 06:27:27.991578 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:27:27.991798 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.986886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:45225 (local address 127.31.214.67:49797)
0928 06:27:27.987080 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:27.987125 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:27.987624 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:27.987864 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:27.987868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:27.987876 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:27.988393 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.988397 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.989185 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.989186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.990113 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:27.990116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.990489 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.990490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:27.990618 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:27.991168 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:27:27.991169 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:27.991198 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:27:27.991222 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:27.991440 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:27.991445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:27.991448 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:27.991504 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:27:27.991601 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260928 06:27:27.992391 19227 heartbeater.cc:347] Connected to a master server at 127.31.214.126:45225
I20260928 06:27:27.992480 19227 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:27.992705 19227 heartbeater.cc:511] Master 127.31.214.126:45225 requested a full tablet report, sending...
I20260928 06:27:27.993166 18777 ts_manager.cc:194] Registered new tserver with Master: e39215a3b9e644c790b4959f5692ef20 (127.31.214.67:34833)
I20260928 06:27:27.993556 18777 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.67:49797
I20260928 06:27:27.994562 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 19099
I20260928 06:27:27.994661 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:27:27.994844 32601 external_mini_cluster.cc:1428] Setting key 609521667ab3d519ad4e8323a0a699b3
I20260928 06:27:27.995857 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:27:28.000703 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:27.997460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.31.214.126:45225)
0928 06:27:27.997503 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:27.997505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:27.997678 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:27:27.997764 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:27.997765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:27.997776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:27.997825 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:27.998022 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.998514 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.999296 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:27.999508 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:27.999582 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.000375 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:28.000376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:28.000377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:28.000382 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:28.000398 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:28.000402 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:28.000522 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:28.000599 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:27:28.000611 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:27:28.004786 18777 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:27:28.006965 18777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:27:28.014577 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.009549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:43537 (local address 127.0.0.1:50158)
0928 06:27:28.010062 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:28.010073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:28.010113 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:28.010420 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:28.010424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:28.010426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:28.010517 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.010521 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.012045 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.012046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.012959 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.012961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.014132 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.014133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.014228 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.014247 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:27:28.014335 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:27:28.014400 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":107,"threads_started":1}
I20260928 06:27:28.014667 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.009530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46043 (local address 127.0.0.1:43148)
0928 06:27:28.009722 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:28.009773 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:28.009846 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:28.010476 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:28.010477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:28.010480 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:28.010517 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.010519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.011869 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.011870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.013086 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.013089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.014134 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.014135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.014228 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.014247 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:27:28.014335 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:27:28.014400 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":29,"thread_start_us":83,"threads_started":1}
I20260928 06:27:28.015209 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.009828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.31.214.66:43537)
0928 06:27:28.010097 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:28.010104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:28.010116 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:27:28.010223 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:28.010226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:28.010243 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:28.010322 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:28.010603 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.011775 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.013057 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.013836 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.013934 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.014062 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:28.014364 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 06:27:28.014408 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20260928 06:27:28.015303 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.009530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:34833 (local address 127.0.0.1:50544)
0928 06:27:28.009722 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:28.009774 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:28.009926 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:28.010549 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:28.010550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:28.010552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:28.010616 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.010617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.011610 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.011611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.013086 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.013089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.015047 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.015048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.015116 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.015121 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:27:28.015178 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:27:28.015225 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":4,"thread_start_us":75,"threads_started":1}
I20260928 06:27:28.015410 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.010044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.31.214.67:34833)
0928 06:27:28.010288 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:28.010293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:28.010305 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:27:28.010354 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:28.010356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:28.010372 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:28.010465 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:28.010701 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.011507 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.014104 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.014858 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.014927 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.015012 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:28.015230 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 06:27:28.015277 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
I20260928 06:27:28.015969 19030 tablet_service.cc:1467] Processing CreateTablet for tablet cccf18fcd8ec4440a1388e1aaf07b14f (DEFAULT_TABLE table=test-table [id=b99e1b218a5b4e65b29cb66628680829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:28.016026 19161 tablet_service.cc:1467] Processing CreateTablet for tablet cccf18fcd8ec4440a1388e1aaf07b14f (DEFAULT_TABLE table=test-table [id=b99e1b218a5b4e65b29cb66628680829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:28.016357 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf18fcd8ec4440a1388e1aaf07b14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:28.016372 19161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf18fcd8ec4440a1388e1aaf07b14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:28.018234 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.009803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.31.214.65:46043)
0928 06:27:28.010299 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:28.010303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:28.010313 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:27:28.010361 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:28.010363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:28.010379 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:28.010594 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:28.010658 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.011776 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.013179 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.013872 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.013936 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.014063 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:28.018065 (+ 4002us) server_negotiation.cc:299] Negotiation successful
0928 06:27:28.018119 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":112,"threads_started":1}
I20260928 06:27:28.018932 18899 tablet_service.cc:1467] Processing CreateTablet for tablet cccf18fcd8ec4440a1388e1aaf07b14f (DEFAULT_TABLE table=test-table [id=b99e1b218a5b4e65b29cb66628680829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:28.019248 19241 tablet_bootstrap.cc:492] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: Bootstrap starting.
I20260928 06:27:28.019292 18899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cccf18fcd8ec4440a1388e1aaf07b14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:28.019778 19242 tablet_bootstrap.cc:492] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: Bootstrap starting.
I20260928 06:27:28.020048 19241 tablet_bootstrap.cc:654] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:28.020354 19241 log.cc:824] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:28.020448 19242 tablet_bootstrap.cc:654] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:28.020869 19242 log.cc:824] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:28.021178 19241 tablet_bootstrap.cc:492] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: No bootstrap required, opened a new log
I20260928 06:27:28.021239 19241 ts_tablet_manager.cc:1397] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:27:28.022615 19245 tablet_bootstrap.cc:492] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d: Bootstrap starting.
I20260928 06:27:28.022789 19242 tablet_bootstrap.cc:492] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: No bootstrap required, opened a new log
I20260928 06:27:28.022859 19242 ts_tablet_manager.cc:1397] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:27:28.023353 19245 tablet_bootstrap.cc:654] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:28.023661 19245 log.cc:824] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:28.024020 19241 raft_consensus.cc:348] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } }
I20260928 06:27:28.024204 19241 raft_consensus.cc:374] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:28.024252 19241 raft_consensus.cc:729] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39215a3b9e644c790b4959f5692ef20, State: Initialized, Role: FOLLOWER
I20260928 06:27:28.024426 19245 tablet_bootstrap.cc:492] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d: No bootstrap required, opened a new log
I20260928 06:27:28.024396 19241 consensus_queue.cc:260] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } }
I20260928 06:27:28.024507 19245 ts_tablet_manager.cc:1397] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:27:28.024674 19241 ts_tablet_manager.cc:1428] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:27:28.024827 19227 heartbeater.cc:503] Master 127.31.214.126:45225 was elected leader, sending a full tablet report...
I20260928 06:27:28.025556 19242 raft_consensus.cc:348] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } }
I20260928 06:27:28.025725 19242 raft_consensus.cc:374] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:28.025774 19242 raft_consensus.cc:729] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d54174252a442aaacbbe155dfc77e9, State: Initialized, Role: FOLLOWER
I20260928 06:27:28.025902 19242 consensus_queue.cc:260] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } }
I20260928 06:27:28.026197 19242 ts_tablet_manager.cc:1428] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:27:28.026206 19096 heartbeater.cc:503] Master 127.31.214.126:45225 was elected leader, sending a full tablet report...
I20260928 06:27:28.027184 19245 raft_consensus.cc:348] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } }
I20260928 06:27:28.027357 19245 raft_consensus.cc:374] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:28.027405 19245 raft_consensus.cc:729] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72389f92a63345fdb69235cca8b90d4d, State: Initialized, Role: FOLLOWER
I20260928 06:27:28.027551 19245 consensus_queue.cc:260] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } }
I20260928 06:27:28.027922 18965 heartbeater.cc:503] Master 127.31.214.126:45225 was elected leader, sending a full tablet report...
I20260928 06:27:28.027977 19245 ts_tablet_manager.cc:1428] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:27:28.058115 19249 raft_consensus.cc:482] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:28.058295 19249 raft_consensus.cc:504] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } }
I20260928 06:27:28.058740 19249 leader_election.cc:288] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e39215a3b9e644c790b4959f5692ef20 (127.31.214.67:34833), 97d54174252a442aaacbbe155dfc77e9 (127.31.214.66:43537)
I20260928 06:27:28.061753 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.058925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:34833 (local address 127.31.214.65:40497)
0928 06:27:28.058997 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:28.059009 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:28.059054 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:28.059221 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:28.059223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:28.059224 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:28.059317 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.059319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.060066 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.060067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.060887 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.060890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.061485 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.061486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.061558 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.061562 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:28.061628 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:27:28.061675 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:27:28.061774 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.059011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:40497 (local address 127.31.214.67:34833)
0928 06:27:28.059070 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:28.059073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:28.059085 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:27:28.059119 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:28.059120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:28.059131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:28.059201 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:28.059414 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.059973 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.060989 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.061410 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.061467 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.061474 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:28.061654 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:27:28.061694 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:27:28.061932 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.058955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:43537 (local address 127.31.214.65:53835)
0928 06:27:28.059203 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:28.059214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:28.059271 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:28.059415 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:28.059415 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:28.059417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:28.059490 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.059492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.060233 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.060234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.061054 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:28.061056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.061664 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.061665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:28.061743 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.061747 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:28.061804 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:27:28.061846 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":163,"threads_started":1}
I20260928 06:27:28.061987 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.059095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:53835 (local address 127.31.214.66:43537)
0928 06:27:28.059151 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:28.059152 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:28.059282 (+ 130us) server_negotiation.cc:407] Connection header received
0928 06:27:28.059329 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:28.059331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:28.059343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:28.059396 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:28.059577 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.060129 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.061155 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.061576 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.061641 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.061648 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:28.061828 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 06:27:28.061879 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:27:28.062197 19181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf18fcd8ec4440a1388e1aaf07b14f" candidate_uuid: "72389f92a63345fdb69235cca8b90d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39215a3b9e644c790b4959f5692ef20" is_pre_election: true
I20260928 06:27:28.062264 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf18fcd8ec4440a1388e1aaf07b14f" candidate_uuid: "72389f92a63345fdb69235cca8b90d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d54174252a442aaacbbe155dfc77e9" is_pre_election: true
I20260928 06:27:28.062376 19181 raft_consensus.cc:2454] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72389f92a63345fdb69235cca8b90d4d in term 0.
I20260928 06:27:28.062395 19050 raft_consensus.cc:2454] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72389f92a63345fdb69235cca8b90d4d in term 0.
I20260928 06:27:28.062637 18853 leader_election.cc:302] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72389f92a63345fdb69235cca8b90d4d, 97d54174252a442aaacbbe155dfc77e9; no voters:
I20260928 06:27:28.062784 19249 raft_consensus.cc:2781] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:27:28.062844 19249 raft_consensus.cc:482] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:27:28.062903 19249 raft_consensus.cc:3037] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:28.063719 19249 raft_consensus.cc:504] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } }
I20260928 06:27:28.063860 19249 leader_election.cc:288] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [CANDIDATE]: Term 1 election: Requested vote from peers e39215a3b9e644c790b4959f5692ef20 (127.31.214.67:34833), 97d54174252a442aaacbbe155dfc77e9 (127.31.214.66:43537)
I20260928 06:27:28.064008 19181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf18fcd8ec4440a1388e1aaf07b14f" candidate_uuid: "72389f92a63345fdb69235cca8b90d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39215a3b9e644c790b4959f5692ef20"
I20260928 06:27:28.064096 19181 raft_consensus.cc:3037] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:28.064113 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cccf18fcd8ec4440a1388e1aaf07b14f" candidate_uuid: "72389f92a63345fdb69235cca8b90d4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97d54174252a442aaacbbe155dfc77e9"
I20260928 06:27:28.064184 19050 raft_consensus.cc:3037] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:28.064836 19181 raft_consensus.cc:2454] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72389f92a63345fdb69235cca8b90d4d in term 1.
I20260928 06:27:28.064867 19050 raft_consensus.cc:2454] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72389f92a63345fdb69235cca8b90d4d in term 1.
I20260928 06:27:28.065018 18854 leader_election.cc:302] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72389f92a63345fdb69235cca8b90d4d, e39215a3b9e644c790b4959f5692ef20; no voters:
I20260928 06:27:28.065207 19249 raft_consensus.cc:2781] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:28.065400 19249 raft_consensus.cc:686] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [term 1 LEADER]: Becoming Leader. State: Replica: 72389f92a63345fdb69235cca8b90d4d, State: Running, Role: LEADER
I20260928 06:27:28.065557 19249 consensus_queue.cc:237] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } }
I20260928 06:27:28.066617 18777 catalog_manager.cc:5495] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 72389f92a63345fdb69235cca8b90d4d (127.31.214.65). New cstate: current_term: 1 leader_uuid: "72389f92a63345fdb69235cca8b90d4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72389f92a63345fdb69235cca8b90d4d" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 46043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 } health_report { overall_health: UNKNOWN } } }
I20260928 06:27:28.074550 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.071900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.31.214.65:46043)
0928 06:27:28.072005 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:28.072006 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:28.072014 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:27:28.072046 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:28.072047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:28.072065 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:28.072136 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:28.072296 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.072783 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.073660 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:28.073853 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:28.073913 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:28.074162 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:28.074253 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:28.074332 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:28.074459 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 06:27:28.074469 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 06:27:28.076610 19181 raft_consensus.cc:1261] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [term 1 FOLLOWER]: Refusing update from remote peer 72389f92a63345fdb69235cca8b90d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:28.076614 19050 raft_consensus.cc:1261] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [term 1 FOLLOWER]: Refusing update from remote peer 72389f92a63345fdb69235cca8b90d4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:28.077005 19249 consensus_queue.cc:1035] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39215a3b9e644c790b4959f5692ef20" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:28.077207 19251 consensus_queue.cc:1035] T cccf18fcd8ec4440a1388e1aaf07b14f P 72389f92a63345fdb69235cca8b90d4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "97d54174252a442aaacbbe155dfc77e9" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:28.080896 19256 mvcc.cc:205] Tried to move back new op lower bound from 7334202769718878208 to 7334202769677275136. Current Snapshot: MvccSnapshot[applied={T|T < 7334202769718878208}]
I20260928 06:27:28.082650 18776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:28.082760 18776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:28.083652 18776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e51b3de4bc8e4836a7285e394eedd9c2: Sending DeleteTablet for 3 replicas of tablet cccf18fcd8ec4440a1388e1aaf07b14f
I20260928 06:27:28.084098 18899 tablet_service.cc:1514] Processing DeleteTablet for tablet cccf18fcd8ec4440a1388e1aaf07b14f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:28 UTC) from {username='slave'} at 127.0.0.1:43148
I20260928 06:27:28.084147 19161 tablet_service.cc:1514] Processing DeleteTablet for tablet cccf18fcd8ec4440a1388e1aaf07b14f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:28 UTC) from {username='slave'} at 127.0.0.1:50544
I20260928 06:27:28.084151 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet cccf18fcd8ec4440a1388e1aaf07b14f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:28 UTC) from {username='slave'} at 127.0.0.1:50158
I20260928 06:27:28.084468 19266 tablet_replica.cc:330] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: stopping tablet replica
I20260928 06:27:28.084580 19266 raft_consensus.cc:2229] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:28.084687 19266 raft_consensus.cc:2258] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:28.084678 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 18837
I20260928 06:27:28.084848 19264 tablet_replica.cc:330] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: stopping tablet replica
I20260928 06:27:28.084951 19264 raft_consensus.cc:2229] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:28.085067 19264 raft_consensus.cc:2258] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:28.085809 19266 ts_tablet_manager.cc:1905] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:28.086894 19264 ts_tablet_manager.cc:1905] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:28.087826 19266 ts_tablet_manager.cc:1918] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:28.087913 19266 log.cc:1196] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/cccf18fcd8ec4440a1388e1aaf07b14f
I20260928 06:27:28.088181 19266 ts_tablet_manager.cc:1939] T cccf18fcd8ec4440a1388e1aaf07b14f P 97d54174252a442aaacbbe155dfc77e9: Deleting consensus metadata
I20260928 06:27:28.088591 19264 ts_tablet_manager.cc:1918] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:28.088670 19264 log.cc:1196] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/cccf18fcd8ec4440a1388e1aaf07b14f
I20260928 06:27:28.088651 18762 catalog_manager.cc:4841] TS 97d54174252a442aaacbbe155dfc77e9 (127.31.214.66:43537): tablet cccf18fcd8ec4440a1388e1aaf07b14f (table test-table [id=b99e1b218a5b4e65b29cb66628680829]) successfully deleted
I20260928 06:27:28.088899 19264 ts_tablet_manager.cc:1939] T cccf18fcd8ec4440a1388e1aaf07b14f P e39215a3b9e644c790b4959f5692ef20: Deleting consensus metadata
I20260928 06:27:28.089361 18763 catalog_manager.cc:4841] TS e39215a3b9e644c790b4959f5692ef20 (127.31.214.67:34833): tablet cccf18fcd8ec4440a1388e1aaf07b14f (table test-table [id=b99e1b218a5b4e65b29cb66628680829]) successfully deleted
W20260928 06:27:28.092329 18761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:46043 (error 108)
I20260928 06:27:28.092401 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 18968
I20260928 06:27:28.093492 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:28.093315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:46043 (local address 127.0.0.1:43164)
0928 06:27:28.093392 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:28.093455 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 06:27:28.093638 18761 catalog_manager.cc:4584] TS 72389f92a63345fdb69235cca8b90d4d (127.31.214.65:46043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cccf18fcd8ec4440a1388e1aaf07b14f: Network error: Client connection negotiation failed: client connection to 127.31.214.65:46043: connect: Connection refused (error 111)
I20260928 06:27:28.098630 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 19099
I20260928 06:27:28.103812 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 18746
2026-09-28T06:27:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (766 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:28 dist-test-slave-q03f krb5kdc[19270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:28 dist-test-slave-q03f krb5kdc[19270](info): set up 2 sockets
Sep 28 06:27:28 dist-test-slave-q03f krb5kdc[19270](info): commencing operation
krb5kdc: starting...
W20260928 06:27:29.230376 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:29 dist-test-slave-q03f krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:29.357714 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35495
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:35495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:29.442904 19283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:29.443171 19283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:29.443213 19283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:29.444376 19283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:29.444445 19283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:29.444470 19283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:29.444489 19283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:29.444521 19283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:29.446070 19283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:35495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:29.446362 19283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:29.446599 19283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:29.451843 19291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:29.451849 19289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:29.451884 19283 server_base.cc:1034] running on GCE node
W20260928 06:27:29.451973 19288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:29.452334 19283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:29.452646 19283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:29.453822 19283 hybrid_clock.cc:648] HybridClock initialized: now 1790576849453788 us; error 59 us; skew 500 ppm
Sep 28 06:27:29 dist-test-slave-q03f krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:29.456598 19283 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:29.457111 19283 webserver.cc:460] Webserver started at http://127.31.214.126:44777/ using document root <none> and password file <none>
I20260928 06:27:29.457309 19283 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:29.457350 19283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:29.457444 19283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:29.458813 19283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "4c81da0d366e4ebea77d3bb0ce4420f4"
format_stamp: "Formatted at 2026-09-28 06:27:29 on dist-test-slave-q03f"
server_key: "7c5ac8038361455831a93e4c8f2bdf51"
server_key_iv: "28890fbfa53ec4d8c8645da3d5f72c3c"
server_key_version: "encryptionkey@0"
I20260928 06:27:29.459156 19283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "4c81da0d366e4ebea77d3bb0ce4420f4"
format_stamp: "Formatted at 2026-09-28 06:27:29 on dist-test-slave-q03f"
server_key: "7c5ac8038361455831a93e4c8f2bdf51"
server_key_iv: "28890fbfa53ec4d8c8645da3d5f72c3c"
server_key_version: "encryptionkey@0"
I20260928 06:27:29.460615 19283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:27:29.461560 19298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:29.461831 19283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:29.461920 19283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "4c81da0d366e4ebea77d3bb0ce4420f4"
format_stamp: "Formatted at 2026-09-28 06:27:29 on dist-test-slave-q03f"
server_key: "7c5ac8038361455831a93e4c8f2bdf51"
server_key_iv: "28890fbfa53ec4d8c8645da3d5f72c3c"
server_key_version: "encryptionkey@0"
I20260928 06:27:29.462122 19283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:29.495177 19283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:29.497506 19283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:29.497658 19283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:29.501998 19283 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:35495
I20260928 06:27:29.502087 19350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:35495 every 8 connection(s)
I20260928 06:27:29.502454 19283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:29.503494 19351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:29.503681 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 19283
I20260928 06:27:29.503819 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:29.504050 32601 external_mini_cluster.cc:1428] Setting key 5670e229a94b6f721b831466a501f57b
I20260928 06:27:29.507591 19351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4: Bootstrap starting.
I20260928 06:27:29.508451 19351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:29.508829 19351 log.cc:824] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:29.509735 19351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4: No bootstrap required, opened a new log
Sep 28 06:27:29 dist-test-slave-q03f krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:29.511966 19351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c81da0d366e4ebea77d3bb0ce4420f4" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35495 } }
I20260928 06:27:29.512176 19351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:29.512328 19351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c81da0d366e4ebea77d3bb0ce4420f4, State: Initialized, Role: FOLLOWER
I20260928 06:27:29.512511 19351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c81da0d366e4ebea77d3bb0ce4420f4" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35495 } }
I20260928 06:27:29.512606 19351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:29.512633 19351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:29.512717 19351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:29.513574 19351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c81da0d366e4ebea77d3bb0ce4420f4" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35495 } }
I20260928 06:27:29.513702 19351 leader_election.cc:302] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c81da0d366e4ebea77d3bb0ce4420f4; no voters:
I20260928 06:27:29.513877 19351 leader_election.cc:288] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:29.514196 19351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:29.514371 19357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:29.514650 19357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [term 1 LEADER]: Becoming Leader. State: Replica: 4c81da0d366e4ebea77d3bb0ce4420f4, State: Running, Role: LEADER
I20260928 06:27:29.514780 19357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c81da0d366e4ebea77d3bb0ce4420f4" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35495 } }
I20260928 06:27:29.515180 19357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c81da0d366e4ebea77d3bb0ce4420f4. Latest consensus state: current_term: 1 leader_uuid: "4c81da0d366e4ebea77d3bb0ce4420f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c81da0d366e4ebea77d3bb0ce4420f4" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35495 } } }
I20260928 06:27:29.515275 19357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:29.515563 19366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:29.516266 19366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:29.516474 19356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c81da0d366e4ebea77d3bb0ce4420f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c81da0d366e4ebea77d3bb0ce4420f4" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35495 } } }
I20260928 06:27:29.516566 19356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:29.517195 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:29.504598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.31.214.126:35495)
0928 06:27:29.504781 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:29.504789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:29.504815 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:27:29.505309 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:29.505316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:29.505366 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:29.505484 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:29.506570 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.507498 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.508706 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.509224 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.509318 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:29.512132 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:29.512136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:29.512136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:29.512150 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:29.514587 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:29.515105 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:29.515109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:29.515109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:29.515233 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:29.515569 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:29.515570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:29.515570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:29.515968 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:29.516169 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:29.516806 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 06:27:29.516990 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260928 06:27:29.518378 19366 catalog_manager.cc:1347] Generated new cluster ID: d7531bf3e37d4829a3f661206b64e12c
I20260928 06:27:29.518442 19366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:29.527992 19366 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:29.528805 19366 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:29.536707 19366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4: Generated new TSK 0
I20260928 06:27:29.537128 19366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.214.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
Entry for principal kudu/127.31.214.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab.
I20260928 06:27:29.573866 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35495
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:29.660086 19377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:29.660311 19377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:29.660337 19377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:29.661516 19377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:29.661574 19377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:29.661603 19377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:27:29.663142 19377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.65.keytab
--principal=kudu/127.31.214.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:29.663406 19377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:29.663671 19377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:27:29.669464 19377 server_base.cc:1034] running on GCE node
W20260928 06:27:29.669589 19383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:29.669500 19385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:29.669485 19382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:29.669919 19377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:29.670223 19377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:29.671396 19377 hybrid_clock.cc:648] HybridClock initialized: now 1790576849671360 us; error 81 us; skew 500 ppm
Sep 28 06:27:29 dist-test-slave-q03f krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:29.674191 19377 init.cc:377] Logged in from keytab as kudu/127.31.214.65@KRBTEST.COM (short username kudu)
I20260928 06:27:29.674645 19377 webserver.cc:460] Webserver started at http://127.31.214.65:40357/ using document root <none> and password file <none>
I20260928 06:27:29.674921 19377 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:29.674969 19377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:29.675113 19377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:29.676245 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "004850f64e1c4193b930c06e4541439e"
format_stamp: "Formatted at 2026-09-28 06:27:29 on dist-test-slave-q03f"
server_key: "e0acdbc0697706a35fa41291c472d736"
server_key_iv: "11c6968fc53daaa1d87452287e406510"
server_key_version: "encryptionkey@0"
I20260928 06:27:29.676628 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "004850f64e1c4193b930c06e4541439e"
format_stamp: "Formatted at 2026-09-28 06:27:29 on dist-test-slave-q03f"
server_key: "e0acdbc0697706a35fa41291c472d736"
server_key_iv: "11c6968fc53daaa1d87452287e406510"
server_key_version: "encryptionkey@0"
I20260928 06:27:29.678207 19377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:27:29.679145 19392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:29.679388 19377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:29.679469 19377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "004850f64e1c4193b930c06e4541439e"
format_stamp: "Formatted at 2026-09-28 06:27:29 on dist-test-slave-q03f"
server_key: "e0acdbc0697706a35fa41291c472d736"
server_key_iv: "11c6968fc53daaa1d87452287e406510"
server_key_version: "encryptionkey@0"
I20260928 06:27:29.679526 19377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:29.690093 19377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:29.692171 19377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:29.692288 19377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:29.692479 19377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:29.692827 19377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:29.692868 19377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:29.692891 19377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:29.692904 19377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:29.699319 19377 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:43343
I20260928 06:27:29.699366 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:43343 every 8 connection(s)
I20260928 06:27:29.699774 19377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:27:29 dist-test-slave-q03f krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.65@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:29.709887 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 19377
I20260928 06:27:29.710083 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:27:29.710319 32601 external_mini_cluster.cc:1428] Setting key ca86f1ea435d2c89758e38bbee58fd1c
I20260928 06:27:29.711285 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:29.700654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:39167 (local address 127.31.214.126:35495)
0928 06:27:29.700743 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:29.700745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:29.701063 (+ 318us) server_negotiation.cc:407] Connection header received
0928 06:27:29.701693 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:29.701694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:29.701705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:29.701766 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:29.703071 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.703685 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.704859 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.705119 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.705194 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:29.707690 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:29.707692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:29.707692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:29.707700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:29.709149 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:29.709601 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:29.709602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:29.709602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:29.709679 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:29.710405 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:29.710407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:29.710407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:29.710608 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:29.710714 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:29.711017 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 06:27:29.711167 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:27:29.711434 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:29.700612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35495 (local address 127.31.214.65:39167)
0928 06:27:29.700822 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:29.700871 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:29.701573 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:29.701884 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:29.701888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:29.702706 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:29.702935 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:29.702942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.703796 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.703797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:29.704747 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:29.704750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.705214 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.705215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:29.705330 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:29.705893 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:29.705902 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:29.707590 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:29.709246 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:29.709253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:29.709255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:29.709496 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:29.710133 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:29.710135 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:29.710135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:29.710299 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:29.710688 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:29.710696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:29.710862 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 06:27:29.710949 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:27:29.711169 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":123,"threads_started":1}
I20260928 06:27:29.712073 19506 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35495
I20260928 06:27:29.712199 19506 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:29.712430 19506 heartbeater.cc:511] Master 127.31.214.126:35495 requested a full tablet report, sending...
I20260928 06:27:29.713147 19315 ts_manager.cc:194] Registered new tserver with Master: 004850f64e1c4193b930c06e4541439e (127.31.214.65:43343)
I20260928 06:27:29.714229 19315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.65@KRBTEST.COM'} at 127.31.214.65:39167
WARNING: no policy specified for kudu/127.31.214.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
Entry for principal kudu/127.31.214.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab.
I20260928 06:27:29.742208 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35495
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:29.832778 19511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:29.833002 19511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:29.833027 19511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:29.834196 19511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:29.834240 19511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:29.834266 19511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:27:29.836318 19511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.66.keytab
--principal=kudu/127.31.214.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:29.836552 19511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:29.836747 19511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:29.842361 19516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:29.842391 19519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:29.842504 19517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:29.842742 19511 server_base.cc:1034] running on GCE node
I20260928 06:27:29.842983 19511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:29.843273 19511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:29.844447 19511 hybrid_clock.cc:648] HybridClock initialized: now 1790576849844423 us; error 52 us; skew 500 ppm
Sep 28 06:27:29 dist-test-slave-q03f krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:29.847187 19511 init.cc:377] Logged in from keytab as kudu/127.31.214.66@KRBTEST.COM (short username kudu)
I20260928 06:27:29.847733 19511 webserver.cc:460] Webserver started at http://127.31.214.66:41137/ using document root <none> and password file <none>
I20260928 06:27:29.847978 19511 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:29.848023 19511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:29.848135 19511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:29.849211 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "131aaf4a423b4944a67a10c721857a5e"
format_stamp: "Formatted at 2026-09-28 06:27:29 on dist-test-slave-q03f"
server_key: "a3623159d94be21d0673db779b18f30f"
server_key_iv: "7b98e0a007a572ec6aa4d07d738c1132"
server_key_version: "encryptionkey@0"
I20260928 06:27:29.849584 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "131aaf4a423b4944a67a10c721857a5e"
format_stamp: "Formatted at 2026-09-28 06:27:29 on dist-test-slave-q03f"
server_key: "a3623159d94be21d0673db779b18f30f"
server_key_iv: "7b98e0a007a572ec6aa4d07d738c1132"
server_key_version: "encryptionkey@0"
I20260928 06:27:29.851053 19511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:27:29.851918 19526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:29.852162 19511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:29.852247 19511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "131aaf4a423b4944a67a10c721857a5e"
format_stamp: "Formatted at 2026-09-28 06:27:29 on dist-test-slave-q03f"
server_key: "a3623159d94be21d0673db779b18f30f"
server_key_iv: "7b98e0a007a572ec6aa4d07d738c1132"
server_key_version: "encryptionkey@0"
I20260928 06:27:29.852304 19511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:29.865351 19511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:29.867971 19511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:29.868093 19511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:29.868355 19511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:29.868650 19511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:29.868683 19511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:29.868707 19511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:29.868719 19511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:29.874943 19511 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:33109
I20260928 06:27:29.875054 19639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:33109 every 8 connection(s)
I20260928 06:27:29.875360 19511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:27:29.878507 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 19511
I20260928 06:27:29.878664 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:27:29.878872 32601 external_mini_cluster.cc:1428] Setting key 89481b73f361c8372c59f15db132d925
Sep 28 06:27:29 dist-test-slave-q03f krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576849, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.66@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:29.886772 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:29.876172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:53151 (local address 127.31.214.126:35495)
0928 06:27:29.876249 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:29.876251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:29.876498 (+ 247us) server_negotiation.cc:407] Connection header received
0928 06:27:29.877082 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:29.877084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:29.877095 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:29.877154 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:29.878231 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.878852 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.880384 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.880724 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.880767 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:29.883563 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:29.883564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:29.883565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:29.883573 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:29.885146 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:29.885626 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:29.885627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:29.885627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:29.885697 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:29.886031 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:29.886032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:29.886032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:29.886163 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:29.886236 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:29.886587 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 06:27:29.886675 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:27:29.887012 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:29.876156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35495 (local address 127.31.214.66:53151)
0928 06:27:29.876352 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:29.876394 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:29.876877 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:29.877244 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:29.877248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:29.877844 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:29.878134 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:29.878140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.878957 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.878958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:29.880266 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:29.880269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:29.880865 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:29.880866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:29.881026 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:29.881688 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:29.881698 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:29.883459 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:29.885273 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:29.885282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:29.885285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:29.885542 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:29.885765 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:29.885766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:29.885767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:29.885912 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:29.886242 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:29.886251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:29.886429 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 06:27:29.886548 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:27:29.886734 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260928 06:27:29.887523 19640 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35495
I20260928 06:27:29.887624 19640 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:29.887825 19640 heartbeater.cc:511] Master 127.31.214.126:35495 requested a full tablet report, sending...
I20260928 06:27:29.888262 19315 ts_manager.cc:194] Registered new tserver with Master: 131aaf4a423b4944a67a10c721857a5e (127.31.214.66:33109)
WARNING: no policy specified for kudu/127.31.214.67@KRBTEST.COM; defaulting to no policy
I20260928 06:27:29.888655 19315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.66@KRBTEST.COM'} at 127.31.214.66:53151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
Entry for principal kudu/127.31.214.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab.
I20260928 06:27:29.910342 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35495
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:29.995220 19645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:29.995448 19645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:29.995476 19645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:29.996618 19645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:29.996659 19645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:29.996686 19645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:27:29.998703 19645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.67.keytab
--principal=kudu/127.31.214.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:35495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:29.999034 19645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:29.999321 19645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:30.006767 19650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:30.006794 19651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:30.006767 19653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:30.007656 19645 server_base.cc:1034] running on GCE node
I20260928 06:27:30.007954 19645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:30.008246 19645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:30.009431 19645 hybrid_clock.cc:648] HybridClock initialized: now 1790576850009399 us; error 49 us; skew 500 ppm
Sep 28 06:27:30 dist-test-slave-q03f krb5kdc[19270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:30.012647 19645 init.cc:377] Logged in from keytab as kudu/127.31.214.67@KRBTEST.COM (short username kudu)
I20260928 06:27:30.013252 19645 webserver.cc:460] Webserver started at http://127.31.214.67:37891/ using document root <none> and password file <none>
I20260928 06:27:30.013550 19645 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:30.013621 19645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:30.013785 19645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:30.015074 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "8e78aa44c69e423b9af719fd50c43048"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "3f89b5598dda8bda78a059bb8e4e8b62"
server_key_iv: "7675ae28ca4dd7981cfb721a2e47788c"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.015491 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "8e78aa44c69e423b9af719fd50c43048"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "3f89b5598dda8bda78a059bb8e4e8b62"
server_key_iv: "7675ae28ca4dd7981cfb721a2e47788c"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.017122 19645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:27:30.018035 19660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.018213 19645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:30.018317 19645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "8e78aa44c69e423b9af719fd50c43048"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "3f89b5598dda8bda78a059bb8e4e8b62"
server_key_iv: "7675ae28ca4dd7981cfb721a2e47788c"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.018409 19645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:30.035775 19645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:30.038551 19645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:30.038746 19645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:30.039011 19645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:30.039530 19645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:30.039604 19645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.039652 19645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:30.039693 19645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.046109 19645 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:39867
I20260928 06:27:30.046182 19773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:39867 every 8 connection(s)
I20260928 06:27:30.046566 19645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:27:30 dist-test-slave-q03f krb5kdc[19270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576850, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.67@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:30.056277 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 19645
I20260928 06:27:30.056449 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:27:30.056635 32601 external_mini_cluster.cc:1428] Setting key 15a39f73a7f0a1f0528a7391a464a148
I20260928 06:27:30.056782 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.047450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:52947 (local address 127.31.214.126:35495)
0928 06:27:30.047538 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.047540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.047693 (+ 153us) server_negotiation.cc:407] Connection header received
0928 06:27:30.048294 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.048296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.048308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.048369 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:30.049399 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.049912 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.051022 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.051290 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.051357 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.053652 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:30.053654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:30.053654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:30.053661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:30.055188 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:30.055643 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:30.055644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:30.055645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:30.055702 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:30.056000 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:30.056001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:30.056001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:30.056117 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:30.056203 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.056590 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.056680 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:27:30.057018 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.047370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:35495 (local address 127.31.214.67:52947)
0928 06:27:30.047537 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.047578 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.048158 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.048473 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.048478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.049083 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:30.049298 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.049303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.050074 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.050075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.050928 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.050930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.051394 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.051395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.051521 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.052076 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:30.052085 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:30.053561 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:30.055296 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:30.055304 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:30.055306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:30.055573 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:30.055801 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:30.055802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:30.055802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:30.055928 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:30.056195 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:30.056204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:30.056401 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.056562 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.056752 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260928 06:27:30.057538 19774 heartbeater.cc:347] Connected to a master server at 127.31.214.126:35495
I20260928 06:27:30.057639 19774 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:30.057860 19774 heartbeater.cc:511] Master 127.31.214.126:35495 requested a full tablet report, sending...
I20260928 06:27:30.058272 19314 ts_manager.cc:194] Registered new tserver with Master: 8e78aa44c69e423b9af719fd50c43048 (127.31.214.67:39867)
I20260928 06:27:30.058635 19314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.214.67@KRBTEST.COM'} at 127.31.214.67:52947
I20260928 06:27:30.068858 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:27:30.077461 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.070464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.31.214.126:35495)
0928 06:27:30.070554 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.070557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.070662 (+ 105us) server_negotiation.cc:407] Connection header received
0928 06:27:30.070772 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.070773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.070785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.070851 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:30.071551 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.072118 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.072952 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.073178 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.073255 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.074705 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:30.074707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:30.074708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:30.074716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:30.076161 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:30.076575 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:30.076577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:30.076577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:30.076642 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:30.076848 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:30.076849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:30.076850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:30.077017 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:30.077127 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.077264 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.077353 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:27:30.079337 19314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:27:30.081084 19314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:27:30.088339 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.083944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33109 (local address 127.0.0.1:54814)
0928 06:27:30.084337 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.084345 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.084381 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.084835 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.084836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.084838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.084888 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.084889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.085906 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.085907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.087161 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.087164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.088005 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.088006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.088086 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.088105 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.088190 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.088251 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":86,"thread_start_us":110,"threads_started":1}
I20260928 06:27:30.088433 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.084115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.31.214.65:43343)
0928 06:27:30.084333 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.084339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.084351 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:27:30.084398 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.084401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.084418 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.084514 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.084756 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.085912 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.087481 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.088063 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.088131 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.088189 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.088296 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.088337 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":103,"threads_started":1}
I20260928 06:27:30.088580 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.083925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:43343 (local address 127.0.0.1:59310)
0928 06:27:30.084116 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.084158 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.084230 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.084538 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.084542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.084544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.084623 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.084625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.086028 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.086029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.087383 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.087386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.088149 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.088151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.088216 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.088220 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.088281 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.088333 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":117,"thread_start_us":53,"threads_started":1}
I20260928 06:27:30.088835 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.083944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:39867 (local address 127.0.0.1:55536)
0928 06:27:30.084308 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.084319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.084367 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.085157 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.085158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.085161 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.085211 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.085212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.086361 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.086362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.087503 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.087505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.088595 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.088596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.088665 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.088670 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.088713 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.088755 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":135,"threads_started":1}
I20260928 06:27:30.088912 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.084237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.31.214.66:33109)
0928 06:27:30.084680 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.084685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.084696 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:27:30.084735 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.084737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.084753 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.084838 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.084968 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.085792 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.087263 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.087909 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.087987 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.088055 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.088776 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.088827 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":90,"threads_started":1}
I20260928 06:27:30.089080 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.084458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.31.214.67:39867)
0928 06:27:30.084974 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.084978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.084987 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:27:30.085025 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.085027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.085055 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.085131 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.085294 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.086266 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.087598 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.088297 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.088357 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.088910 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.088943 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.088994 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":84,"threads_started":1}
I20260928 06:27:30.089305 19440 tablet_service.cc:1467] Processing CreateTablet for tablet 622c05f2f92c425c93644b7e52d76f84 (DEFAULT_TABLE table=test-table [id=1431836513ea4451b696e46b4dec09ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:30.089589 19440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622c05f2f92c425c93644b7e52d76f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:30.092330 19574 tablet_service.cc:1467] Processing CreateTablet for tablet 622c05f2f92c425c93644b7e52d76f84 (DEFAULT_TABLE table=test-table [id=1431836513ea4451b696e46b4dec09ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:30.092540 19708 tablet_service.cc:1467] Processing CreateTablet for tablet 622c05f2f92c425c93644b7e52d76f84 (DEFAULT_TABLE table=test-table [id=1431836513ea4451b696e46b4dec09ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:30.092715 19574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622c05f2f92c425c93644b7e52d76f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:30.092751 19791 tablet_bootstrap.cc:492] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e: Bootstrap starting.
I20260928 06:27:30.092871 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622c05f2f92c425c93644b7e52d76f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:30.093422 19791 tablet_bootstrap.cc:654] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:30.094038 19791 log.cc:824] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:30.094841 19791 tablet_bootstrap.cc:492] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e: No bootstrap required, opened a new log
I20260928 06:27:30.094972 19791 ts_tablet_manager.cc:1397] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:27:30.096230 19793 tablet_bootstrap.cc:492] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: Bootstrap starting.
I20260928 06:27:30.096495 19794 tablet_bootstrap.cc:492] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: Bootstrap starting.
I20260928 06:27:30.096987 19793 tablet_bootstrap.cc:654] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:30.097113 19794 tablet_bootstrap.cc:654] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:30.097604 19794 log.cc:824] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:30.097615 19793 log.cc:824] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:30.097683 19791 raft_consensus.cc:348] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } }
I20260928 06:27:30.097980 19791 raft_consensus.cc:374] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:30.098135 19791 raft_consensus.cc:729] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004850f64e1c4193b930c06e4541439e, State: Initialized, Role: FOLLOWER
I20260928 06:27:30.098274 19791 consensus_queue.cc:260] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } }
I20260928 06:27:30.098492 19794 tablet_bootstrap.cc:492] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: No bootstrap required, opened a new log
I20260928 06:27:30.098564 19794 ts_tablet_manager.cc:1397] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:27:30.098582 19791 ts_tablet_manager.cc:1428] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 06:27:30.098592 19506 heartbeater.cc:503] Master 127.31.214.126:35495 was elected leader, sending a full tablet report...
I20260928 06:27:30.099035 19793 tablet_bootstrap.cc:492] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: No bootstrap required, opened a new log
I20260928 06:27:30.099104 19793 ts_tablet_manager.cc:1397] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:27:30.100617 19794 raft_consensus.cc:348] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } }
I20260928 06:27:30.100803 19794 raft_consensus.cc:374] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:30.100837 19794 raft_consensus.cc:729] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e78aa44c69e423b9af719fd50c43048, State: Initialized, Role: FOLLOWER
I20260928 06:27:30.100951 19794 consensus_queue.cc:260] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } }
I20260928 06:27:30.101075 19793 raft_consensus.cc:348] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } }
I20260928 06:27:30.101250 19793 raft_consensus.cc:374] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:30.101261 19794 ts_tablet_manager.cc:1428] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:27:30.101300 19793 raft_consensus.cc:729] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131aaf4a423b4944a67a10c721857a5e, State: Initialized, Role: FOLLOWER
I20260928 06:27:30.101306 19774 heartbeater.cc:503] Master 127.31.214.126:35495 was elected leader, sending a full tablet report...
I20260928 06:27:30.101418 19793 consensus_queue.cc:260] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } }
I20260928 06:27:30.101732 19793 ts_tablet_manager.cc:1428] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:27:30.101871 19640 heartbeater.cc:503] Master 127.31.214.126:35495 was elected leader, sending a full tablet report...
W20260928 06:27:30.126119 19641 tablet.cc:2369] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:27:30.200618 19507 tablet.cc:2369] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:27:30.226900 19797 raft_consensus.cc:482] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:30.227037 19797 raft_consensus.cc:504] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } }
I20260928 06:27:30.227396 19797 leader_election.cc:288] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e78aa44c69e423b9af719fd50c43048 (127.31.214.67:39867), 131aaf4a423b4944a67a10c721857a5e (127.31.214.66:33109)
I20260928 06:27:30.230626 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.227738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:52311 (local address 127.31.214.66:33109)
0928 06:27:30.227788 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.227790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.227901 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:27:30.227970 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.227971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.227984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.228112 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.228311 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.228779 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.229802 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.230259 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.230347 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.230356 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.230505 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.230546 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:27:30.230625 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.227713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:49955 (local address 127.31.214.67:39867)
0928 06:27:30.227764 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.227767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.227961 (+ 194us) server_negotiation.cc:407] Connection header received
0928 06:27:30.227999 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.228000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.228008 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.228074 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.228247 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.228755 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.229790 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.230266 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.230356 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.230365 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.230515 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.230554 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:27:30.230650 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.227607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:39867 (local address 127.31.214.65:49955)
0928 06:27:30.227789 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.227802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.227875 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.228079 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.228080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.228082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.228167 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.228169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.228855 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.228856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.229692 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.229694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.230373 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.230375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.230443 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.230447 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.230510 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.230555 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260928 06:27:30.230759 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.227654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:33109 (local address 127.31.214.65:52311)
0928 06:27:30.227814 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.227823 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.227908 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.228132 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.228133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.228135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.228227 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.228229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.228867 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.228868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.229703 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.229706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.230345 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.230346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.230426 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.230430 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.230497 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.230549 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260928 06:27:30.231010 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622c05f2f92c425c93644b7e52d76f84" candidate_uuid: "004850f64e1c4193b930c06e4541439e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78aa44c69e423b9af719fd50c43048" is_pre_election: true
I20260928 06:27:30.231172 19728 raft_consensus.cc:2454] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 004850f64e1c4193b930c06e4541439e in term 0.
I20260928 06:27:30.231241 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622c05f2f92c425c93644b7e52d76f84" candidate_uuid: "004850f64e1c4193b930c06e4541439e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "131aaf4a423b4944a67a10c721857a5e" is_pre_election: true
I20260928 06:27:30.231395 19594 raft_consensus.cc:2454] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 004850f64e1c4193b930c06e4541439e in term 0.
I20260928 06:27:30.231455 19396 leader_election.cc:302] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004850f64e1c4193b930c06e4541439e, 8e78aa44c69e423b9af719fd50c43048; no voters:
I20260928 06:27:30.231595 19797 raft_consensus.cc:2781] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:27:30.231658 19797 raft_consensus.cc:482] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:27:30.231683 19797 raft_consensus.cc:3037] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:30.232443 19797 raft_consensus.cc:504] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } }
I20260928 06:27:30.232610 19797 leader_election.cc:288] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [CANDIDATE]: Term 1 election: Requested vote from peers 8e78aa44c69e423b9af719fd50c43048 (127.31.214.67:39867), 131aaf4a423b4944a67a10c721857a5e (127.31.214.66:33109)
I20260928 06:27:30.232779 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622c05f2f92c425c93644b7e52d76f84" candidate_uuid: "004850f64e1c4193b930c06e4541439e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e78aa44c69e423b9af719fd50c43048"
I20260928 06:27:30.232807 19594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622c05f2f92c425c93644b7e52d76f84" candidate_uuid: "004850f64e1c4193b930c06e4541439e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "131aaf4a423b4944a67a10c721857a5e"
I20260928 06:27:30.232867 19728 raft_consensus.cc:3037] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:30.232897 19594 raft_consensus.cc:3037] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:30.233626 19728 raft_consensus.cc:2454] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 004850f64e1c4193b930c06e4541439e in term 1.
I20260928 06:27:30.233690 19594 raft_consensus.cc:2454] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 004850f64e1c4193b930c06e4541439e in term 1.
I20260928 06:27:30.233799 19396 leader_election.cc:302] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004850f64e1c4193b930c06e4541439e, 8e78aa44c69e423b9af719fd50c43048; no voters:
I20260928 06:27:30.233912 19797 raft_consensus.cc:2781] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:30.234124 19797 raft_consensus.cc:686] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [term 1 LEADER]: Becoming Leader. State: Replica: 004850f64e1c4193b930c06e4541439e, State: Running, Role: LEADER
I20260928 06:27:30.234233 19797 consensus_queue.cc:237] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } }
I20260928 06:27:30.235356 19314 catalog_manager.cc:5495] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e reported cstate change: term changed from 0 to 1, leader changed from <none> to 004850f64e1c4193b930c06e4541439e (127.31.214.65). New cstate: current_term: 1 leader_uuid: "004850f64e1c4193b930c06e4541439e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "004850f64e1c4193b930c06e4541439e" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 43343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 } health_report { overall_health: UNKNOWN } } }
I20260928 06:27:30.272588 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.269746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.31.214.65:43343)
0928 06:27:30.269817 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.269820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.269829 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:27:30.269880 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.269881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.269902 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.270027 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:30.270217 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.270882 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.271698 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.271901 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.271977 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.272219 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:30.272354 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:30.272453 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.272495 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.272519 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:27:30.274792 19594 raft_consensus.cc:1261] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [term 1 FOLLOWER]: Refusing update from remote peer 004850f64e1c4193b930c06e4541439e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:30.274791 19728 raft_consensus.cc:1261] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [term 1 FOLLOWER]: Refusing update from remote peer 004850f64e1c4193b930c06e4541439e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:30.275239 19797 consensus_queue.cc:1035] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [LEADER]: Connected to new peer: Peer: permanent_uuid: "131aaf4a423b4944a67a10c721857a5e" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 33109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:30.275488 19802 consensus_queue.cc:1035] T 622c05f2f92c425c93644b7e52d76f84 P 004850f64e1c4193b930c06e4541439e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e78aa44c69e423b9af719fd50c43048" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 39867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:30.278540 19807 mvcc.cc:205] Tried to move back new op lower bound from 7334202778722222080 to 7334202778560159744. Current Snapshot: MvccSnapshot[applied={T|T < 7334202778722222080}]
I20260928 06:27:30.279156 19805 mvcc.cc:205] Tried to move back new op lower bound from 7334202778722222080 to 7334202778560159744. Current Snapshot: MvccSnapshot[applied={T|T < 7334202778722222080 or (T in {7334202778722222080})}]
I20260928 06:27:30.279552 19806 mvcc.cc:205] Tried to move back new op lower bound from 7334202778722222080 to 7334202778560159744. Current Snapshot: MvccSnapshot[applied={T|T < 7334202778722222080 or (T in {7334202778722222080})}]
I20260928 06:27:30.280827 19314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:30.280936 19314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:30.281738 19314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c81da0d366e4ebea77d3bb0ce4420f4: Sending DeleteTablet for 3 replicas of tablet 622c05f2f92c425c93644b7e52d76f84
I20260928 06:27:30.282271 19708 tablet_service.cc:1514] Processing DeleteTablet for tablet 622c05f2f92c425c93644b7e52d76f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:30 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:55536
I20260928 06:27:30.282259 19440 tablet_service.cc:1514] Processing DeleteTablet for tablet 622c05f2f92c425c93644b7e52d76f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:30 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:59310
I20260928 06:27:30.282393 19574 tablet_service.cc:1514] Processing DeleteTablet for tablet 622c05f2f92c425c93644b7e52d76f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:30 UTC) from {username='kudu', principal='kudu/127.31.214.126@KRBTEST.COM'} at 127.0.0.1:54814
I20260928 06:27:30.282560 19815 tablet_replica.cc:330] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: stopping tablet replica
I20260928 06:27:30.282666 19815 raft_consensus.cc:2229] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:30.282774 19815 raft_consensus.cc:2258] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:30.282982 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 19377
I20260928 06:27:30.283290 19815 ts_tablet_manager.cc:1905] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:30.285607 19815 ts_tablet_manager.cc:1918] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:30.285693 19815 log.cc:1196] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/622c05f2f92c425c93644b7e52d76f84
I20260928 06:27:30.286006 19815 ts_tablet_manager.cc:1939] T 622c05f2f92c425c93644b7e52d76f84 P 8e78aa44c69e423b9af719fd50c43048: Deleting consensus metadata
I20260928 06:27:30.286525 19302 catalog_manager.cc:4841] TS 8e78aa44c69e423b9af719fd50c43048 (127.31.214.67:39867): tablet 622c05f2f92c425c93644b7e52d76f84 (table test-table [id=1431836513ea4451b696e46b4dec09ad]) successfully deleted
I20260928 06:27:30.287142 19817 tablet_replica.cc:330] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: stopping tablet replica
I20260928 06:27:30.287272 19817 raft_consensus.cc:2229] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:30.287395 19817 raft_consensus.cc:2258] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:30.287843 19817 ts_tablet_manager.cc:1905] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:30.289271 19817 ts_tablet_manager.cc:1918] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:30.289368 19817 log.cc:1196] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790576570121568-32601-0/minicluster-data/ts-1/wal/wals/622c05f2f92c425c93644b7e52d76f84
I20260928 06:27:30.289637 19817 ts_tablet_manager.cc:1939] T 622c05f2f92c425c93644b7e52d76f84 P 131aaf4a423b4944a67a10c721857a5e: Deleting consensus metadata
I20260928 06:27:30.290161 19300 catalog_manager.cc:4841] TS 131aaf4a423b4944a67a10c721857a5e (127.31.214.66:33109): tablet 622c05f2f92c425c93644b7e52d76f84 (table test-table [id=1431836513ea4451b696e46b4dec09ad]) successfully deleted
W20260928 06:27:30.291447 19299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:43343 (error 108)
I20260928 06:27:30.291787 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 19511
I20260928 06:27:30.292025 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.291901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:43343 (local address 127.0.0.1:59328)
0928 06:27:30.291951 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.291986 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:43343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:27:30.292132 19299 catalog_manager.cc:4584] TS 004850f64e1c4193b930c06e4541439e (127.31.214.65:43343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 622c05f2f92c425c93644b7e52d76f84: Network error: Client connection negotiation failed: client connection to 127.31.214.65:43343: connect: Connection refused (error 111)
I20260928 06:27:30.297777 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 19645
I20260928 06:27:30.302929 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 19283
2026-09-28T06:27:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2199 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T06:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:30Z Disabled control of system clock
I20260928 06:27:30.332896 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:43493
--webserver_interface=127.31.214.126
--webserver_port=0
--builtin_ntp_servers=127.31.214.84:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:43493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:27:30.425295 19822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:30.425544 19822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:30.425567 19822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:30.426939 19822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:30.426993 19822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:30.427007 19822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:30.427022 19822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:30.427084 19822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:30.428752 19822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:43493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:43493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:30.429075 19822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:30.429435 19822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:30.435034 19830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:30.435163 19828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:30.435111 19827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:30.435252 19822 server_base.cc:1034] running on GCE node
I20260928 06:27:30.435541 19822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:30.435877 19822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:30.437029 19822 hybrid_clock.cc:648] HybridClock initialized: now 1790576850436993 us; error 58 us; skew 500 ppm
I20260928 06:27:30.438491 19822 webserver.cc:460] Webserver started at http://127.31.214.126:34709/ using document root <none> and password file <none>
I20260928 06:27:30.438730 19822 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:30.438774 19822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:30.438870 19822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:30.439947 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "b73588517b964b3da5df75cc4f20feda"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "be7617fa96d0b76362f90461cc15af1b"
server_key_iv: "c4a9236d14ca790ac9ccf64264ff703a"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.440253 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "b73588517b964b3da5df75cc4f20feda"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "be7617fa96d0b76362f90461cc15af1b"
server_key_iv: "c4a9236d14ca790ac9ccf64264ff703a"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.441569 19822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:27:30.442449 19836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.442680 19822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:27:30.442775 19822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "b73588517b964b3da5df75cc4f20feda"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "be7617fa96d0b76362f90461cc15af1b"
server_key_iv: "c4a9236d14ca790ac9ccf64264ff703a"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.442838 19822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:30.453488 19822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:30.453781 19822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:30.453897 19822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:30.458300 19822 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:43493
I20260928 06:27:30.458402 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:43493 every 8 connection(s)
I20260928 06:27:30.458778 19822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:30.459892 19889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:30.462455 19889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda: Bootstrap starting.
I20260928 06:27:30.463081 19889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:30.463419 19889 log.cc:824] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:30.464254 19889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda: No bootstrap required, opened a new log
I20260928 06:27:30.466164 19889 raft_consensus.cc:348] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73588517b964b3da5df75cc4f20feda" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43493 } }
I20260928 06:27:30.466331 19889 raft_consensus.cc:374] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:30.466382 19889 raft_consensus.cc:729] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73588517b964b3da5df75cc4f20feda, State: Initialized, Role: FOLLOWER
I20260928 06:27:30.466563 19889 consensus_queue.cc:260] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73588517b964b3da5df75cc4f20feda" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43493 } }
I20260928 06:27:30.466650 19889 raft_consensus.cc:388] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:30.466696 19889 raft_consensus.cc:482] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:30.466746 19889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:30.467332 19889 raft_consensus.cc:504] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73588517b964b3da5df75cc4f20feda" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43493 } }
I20260928 06:27:30.467458 19889 leader_election.cc:302] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b73588517b964b3da5df75cc4f20feda; no voters:
I20260928 06:27:30.467653 19889 leader_election.cc:288] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:30.467715 19892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:30.467947 19889 sys_catalog.cc:564] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:30.468057 19892 raft_consensus.cc:686] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [term 1 LEADER]: Becoming Leader. State: Replica: b73588517b964b3da5df75cc4f20feda, State: Running, Role: LEADER
I20260928 06:27:30.468080 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 19822
I20260928 06:27:30.468178 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:30.468199 19892 consensus_queue.cc:237] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73588517b964b3da5df75cc4f20feda" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43493 } }
I20260928 06:27:30.468380 32601 external_mini_cluster.cc:1428] Setting key 945c3dd0bcfa9d4948d32e4be63f8531
I20260928 06:27:30.469163 19893 sys_catalog.cc:455] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b73588517b964b3da5df75cc4f20feda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73588517b964b3da5df75cc4f20feda" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43493 } } }
I20260928 06:27:30.469249 19893 sys_catalog.cc:458] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:30.469383 19894 sys_catalog.cc:455] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [sys.catalog]: SysCatalogTable state changed. Reason: New leader b73588517b964b3da5df75cc4f20feda. Latest consensus state: current_term: 1 leader_uuid: "b73588517b964b3da5df75cc4f20feda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73588517b964b3da5df75cc4f20feda" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 43493 } } }
I20260928 06:27:30.469488 19894 sys_catalog.cc:458] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:30.469640 19904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:30.470258 19904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:30.471923 19904 catalog_manager.cc:1347] Generated new cluster ID: b49641b95a5d4180bd5854cccaf6d82d
I20260928 06:27:30.471990 19904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:30.474658 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.468894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.31.214.126:43493)
0928 06:27:30.469245 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.469254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.469288 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:27:30.469415 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.469423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.469536 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.469711 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:30.469976 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.471416 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.472421 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.472991 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.473146 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.473928 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:30.473932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:30.473932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:30.474159 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:30.474193 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:30.474205 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:30.474286 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.474359 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.474443 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":76,"threads_started":1}
I20260928 06:27:30.502760 19904 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:30.503515 19904 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:30.514103 19904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda: Generated new TSK 0
I20260928 06:27:30.514355 19904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:30.523514 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.65:0
--local_ip_for_outbound_sockets=127.31.214.65
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43493
--builtin_ntp_servers=127.31.214.84:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:27:30.611060 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:30.611325 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:30.611372 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:30.612574 19913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:30.612641 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:30.612684 19913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.65
I20260928 06:27:30.614114 19913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.214.65
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.65
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:30.614403 19913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:30.614792 19913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:30.620527 19921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:30.620573 19913 server_base.cc:1034] running on GCE node
W20260928 06:27:30.620544 19919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:30.620541 19918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:30.621038 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:30.621260 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:30.622428 19913 hybrid_clock.cc:648] HybridClock initialized: now 1790576850622405 us; error 50 us; skew 500 ppm
I20260928 06:27:30.623910 19913 webserver.cc:460] Webserver started at http://127.31.214.65:41425/ using document root <none> and password file <none>
I20260928 06:27:30.624241 19913 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:30.624315 19913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:30.624526 19913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:30.625833 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/data/instance:
uuid: "99fbc4a5b3e74db7a80c061a08be553b"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "c639b9da052fa6bf9d0adee3dc289a1f"
server_key_iv: "298f73655877aa334bd113cb66a8e1eb"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.626300 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance:
uuid: "99fbc4a5b3e74db7a80c061a08be553b"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "c639b9da052fa6bf9d0adee3dc289a1f"
server_key_iv: "298f73655877aa334bd113cb66a8e1eb"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.627736 19913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:30.628626 19927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.628867 19913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:30.628975 19913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/wal
uuid: "99fbc4a5b3e74db7a80c061a08be553b"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "c639b9da052fa6bf9d0adee3dc289a1f"
server_key_iv: "298f73655877aa334bd113cb66a8e1eb"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.629051 19913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:30.636799 19913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:30.637171 19913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:30.637291 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:30.637558 19913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:30.638018 19913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:30.638078 19913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.638120 19913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:30.638149 19913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.644775 19913 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:36171
I20260928 06:27:30.644827 20040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:36171 every 8 connection(s)
I20260928 06:27:30.645212 19913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/data/info.pb
I20260928 06:27:30.649355 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 19913
I20260928 06:27:30.649484 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-0/wal/instance
I20260928 06:27:30.649675 32601 external_mini_cluster.cc:1428] Setting key ec1393f02f058c95b720f4c9f602b035
I20260928 06:27:30.650825 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.66:0
--local_ip_for_outbound_sockets=127.31.214.66
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43493
--builtin_ntp_servers=127.31.214.84:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:27:30.651144 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.646062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:37625 (local address 127.31.214.126:43493)
0928 06:27:30.646109 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.646112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.646441 (+ 329us) server_negotiation.cc:407] Connection header received
0928 06:27:30.647014 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.647015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.647031 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.647102 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:30.647843 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.648379 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.649428 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.649706 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.649797 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.650852 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:30.650854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:30.650854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:30.650859 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:30.650874 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:30.650878 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:30.650927 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.651029 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.651065 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:27:30.651382 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.646095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43493 (local address 127.31.214.65:37625)
0928 06:27:30.646288 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.646326 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.646898 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.647213 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.647216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.647224 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:30.647749 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.647753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.648484 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.648485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.649331 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.649333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.649809 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.649810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.650018 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.650697 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:27:30.650697 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:30.650728 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:27:30.650754 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:30.650936 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:30.650941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:30.650945 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.651039 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.651147 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260928 06:27:30.651973 20041 heartbeater.cc:347] Connected to a master server at 127.31.214.126:43493
I20260928 06:27:30.652086 20041 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:30.652315 20041 heartbeater.cc:511] Master 127.31.214.126:43493 requested a full tablet report, sending...
I20260928 06:27:30.652935 19853 ts_manager.cc:194] Registered new tserver with Master: 99fbc4a5b3e74db7a80c061a08be553b (127.31.214.65:36171)
I20260928 06:27:30.653896 19853 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.65:37625
W20260928 06:27:30.735869 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:30.736109 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:30.736167 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:30.737393 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:30.737439 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:30.737466 20044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.66
I20260928 06:27:30.738878 20044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.214.66
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.66
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:30.739127 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:30.739406 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:30.745024 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:30.744975 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:30.745077 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:30.745824 20044 server_base.cc:1034] running on GCE node
I20260928 06:27:30.746151 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:30.746502 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:30.747684 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790576850747667 us; error 50 us; skew 500 ppm
I20260928 06:27:30.749265 20044 webserver.cc:460] Webserver started at http://127.31.214.66:42957/ using document root <none> and password file <none>
I20260928 06:27:30.749562 20044 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:30.749612 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:30.749822 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:30.751036 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/data/instance:
uuid: "4cc5434e77984c39a15a1f4f6a264b7f"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "9d6d958db39dd4f1d5648a2e2f55e5e7"
server_key_iv: "9969b5743146274b27aae9bc96f64cc6"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.751425 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance:
uuid: "4cc5434e77984c39a15a1f4f6a264b7f"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "9d6d958db39dd4f1d5648a2e2f55e5e7"
server_key_iv: "9969b5743146274b27aae9bc96f64cc6"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.752820 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:27:30.753571 20058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.753741 20044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:27:30.753850 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/wal
uuid: "4cc5434e77984c39a15a1f4f6a264b7f"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "9d6d958db39dd4f1d5648a2e2f55e5e7"
server_key_iv: "9969b5743146274b27aae9bc96f64cc6"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.753930 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:30.769520 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:30.769904 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:30.770087 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:30.770334 20044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:30.770701 20044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:30.770759 20044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.770804 20044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:30.770829 20044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.777186 20044 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.66:36851
I20260928 06:27:30.777228 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.66:36851 every 8 connection(s)
I20260928 06:27:30.777591 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/data/info.pb
I20260928 06:27:30.783077 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.778536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.66:42459 (local address 127.31.214.126:43493)
0928 06:27:30.778621 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.778623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.778895 (+ 272us) server_negotiation.cc:407] Connection header received
0928 06:27:30.779384 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.779385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.779396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.779446 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:30.780183 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.780679 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.781692 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.781873 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.781924 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.782848 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:30.782849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:30.782849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:30.782853 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:30.782865 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:30.782869 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:30.782911 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.782980 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.783003 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:27:30.783325 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.778502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43493 (local address 127.31.214.66:42459)
0928 06:27:30.778769 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.778805 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.779301 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.779514 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.779517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.779526 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:30.780083 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.780088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.780767 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.780768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.781610 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.781613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.781968 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.781969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.782087 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.782713 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:27:30.782714 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:30.782743 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:27:30.782768 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:30.782913 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:30.782915 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:30.782917 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.782967 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.783066 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":134,"threads_started":1}
I20260928 06:27:30.783946 20172 heartbeater.cc:347] Connected to a master server at 127.31.214.126:43493
I20260928 06:27:30.784050 20172 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:30.784211 20172 heartbeater.cc:511] Master 127.31.214.126:43493 requested a full tablet report, sending...
I20260928 06:27:30.784618 19853 ts_manager.cc:194] Registered new tserver with Master: 4cc5434e77984c39a15a1f4f6a264b7f (127.31.214.66:36851)
I20260928 06:27:30.784957 19853 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.66:42459
I20260928 06:27:30.787099 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 20044
I20260928 06:27:30.787197 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-1/wal/instance
I20260928 06:27:30.787432 32601 external_mini_cluster.cc:1428] Setting key b747bfa799b7fedbff4ea004057fcfcd
I20260928 06:27:30.788415 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.214.67:0
--local_ip_for_outbound_sockets=127.31.214.67
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43493
--builtin_ntp_servers=127.31.214.84:41149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:27:30.873157 20175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:30.873524 20175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:30.873593 20175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:30.874878 20175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:30.874959 20175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:30.875005 20175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.214.67
I20260928 06:27:30.876543 20175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:41149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.214.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.214.67
--webserver_port=0
--tserver_master_addrs=127.31.214.126:43493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.214.67
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:30.876847 20175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:30.877230 20175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:30.882826 20181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:30.882910 20183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:30.882826 20180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:30.883248 20175 server_base.cc:1034] running on GCE node
I20260928 06:27:30.883483 20175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:30.883744 20175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:30.885012 20175 hybrid_clock.cc:648] HybridClock initialized: now 1790576850884970 us; error 54 us; skew 500 ppm
I20260928 06:27:30.886593 20175 webserver.cc:460] Webserver started at http://127.31.214.67:36711/ using document root <none> and password file <none>
I20260928 06:27:30.886896 20175 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:30.886974 20175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:30.887132 20175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:30.888274 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/data/instance:
uuid: "825b4b574724490da661c3922b759abe"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "7b9a127df18d53789568806696f24236"
server_key_iv: "e54cbde484818fc56e5cc85ef73b933e"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.888643 20175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance:
uuid: "825b4b574724490da661c3922b759abe"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "7b9a127df18d53789568806696f24236"
server_key_iv: "e54cbde484818fc56e5cc85ef73b933e"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.890110 20175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:27:30.890935 20189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.891130 20175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:30.891232 20175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/wal
uuid: "825b4b574724490da661c3922b759abe"
format_stamp: "Formatted at 2026-09-28 06:27:30 on dist-test-slave-q03f"
server_key: "7b9a127df18d53789568806696f24236"
server_key_iv: "e54cbde484818fc56e5cc85ef73b933e"
server_key_version: "encryptionkey@0"
I20260928 06:27:30.891319 20175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:30.908208 20175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:30.908576 20175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:30.908743 20175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:30.908999 20175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:27:30.909356 20175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:27:30.909411 20175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.909447 20175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:27:30.909502 20175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:30.915511 20175 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.67:37271
I20260928 06:27:30.915581 20302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.67:37271 every 8 connection(s)
I20260928 06:27:30.915912 20175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/data/info.pb
I20260928 06:27:30.921325 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.916762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.67:35317 (local address 127.31.214.126:43493)
0928 06:27:30.916827 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.916829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.917100 (+ 271us) server_negotiation.cc:407] Connection header received
0928 06:27:30.917563 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.917564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.917576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.917649 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:30.918345 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.918856 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.919906 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.920102 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.920160 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.921106 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:30.921106 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:30.921107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:30.921111 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:30.921123 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:30.921127 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:30.921169 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.921236 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.921263 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:27:30.921542 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.916740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:43493 (local address 127.31.214.67:35317)
0928 06:27:30.916923 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.916966 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.917456 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.917704 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.917708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.917716 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:30.918258 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.918264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.918947 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.918948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.919815 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.919818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.920187 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.920188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.920304 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.920934 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:27:30.920935 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:30.920965 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:27:30.920991 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:30.921169 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:30.921172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:30.921175 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.921224 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.921308 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 06:27:30.922196 20303 heartbeater.cc:347] Connected to a master server at 127.31.214.126:43493
I20260928 06:27:30.922291 20303 heartbeater.cc:464] Registering TS with master...
I20260928 06:27:30.922446 20303 heartbeater.cc:511] Master 127.31.214.126:43493 requested a full tablet report, sending...
I20260928 06:27:30.922866 19853 ts_manager.cc:194] Registered new tserver with Master: 825b4b574724490da661c3922b759abe (127.31.214.67:37271)
I20260928 06:27:30.923210 19853 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.214.67:35317
I20260928 06:27:30.923900 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 20175
I20260928 06:27:30.923980 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/wal/instance
I20260928 06:27:30.924250 32601 external_mini_cluster.cc:1428] Setting key 51b03857dba77952bf42aa4cbcd8681c
I20260928 06:27:30.925197 32601 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:27:30.929740 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.926720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.31.214.126:43493)
0928 06:27:30.926789 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.926791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.926945 (+ 154us) server_negotiation.cc:407] Connection header received
0928 06:27:30.926981 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.926982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.926994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.927040 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:30.927196 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.927647 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.928421 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.928597 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.928649 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.929457 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:30.929458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:30.929459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:30.929462 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:30.929474 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:30.929478 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:30.929520 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.929603 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.929624 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:27:30.931598 19853 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:27:30.933380 19853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:27:30.940618 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.935864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:36851 (local address 127.0.0.1:52324)
0928 06:27:30.936172 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.936181 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.936222 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.936553 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.936556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.936559 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.936651 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.936654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.938162 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.938163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.939359 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.939362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.940244 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.940245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.940340 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.940358 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.940431 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.940493 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":72,"thread_start_us":96,"threads_started":1}
I20260928 06:27:30.940641 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.936308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.31.214.65:36171)
0928 06:27:30.936510 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.936514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.936841 (+ 327us) server_negotiation.cc:407] Connection header received
0928 06:27:30.936894 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.936896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.936913 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.937002 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.937275 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.938420 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.939681 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.940289 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.940360 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.940476 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.940506 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.940547 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260928 06:27:30.940680 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.936060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.31.214.66:36851)
0928 06:27:30.936367 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.936371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.936380 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:27:30.936419 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.936420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.936437 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.936529 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.936757 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.938075 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.939447 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.940138 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.940220 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.940321 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.940547 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.940590 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260928 06:27:30.940806 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.935863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:37271 (local address 127.0.0.1:41248)
0928 06:27:30.936072 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.936112 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.936183 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.936557 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.936557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.936562 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.936651 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.936653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.937826 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.937827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.939269 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.939272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.940246 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.940247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.940345 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.940359 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.940409 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.940489 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":26,"thread_start_us":77,"threads_started":1}
I20260928 06:27:30.941347 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.936194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.31.214.67:37271)
0928 06:27:30.936361 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.936367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.936377 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:27:30.936418 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.936420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.936437 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.936557 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.936863 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.937739 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.939355 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.940138 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.941028 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.941131 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.941156 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.941264 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260928 06:27:30.940618 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.935867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:36171 (local address 127.0.0.1:59030)
0928 06:27:30.936698 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.936708 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.936761 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.937139 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.937140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.937143 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.937185 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.937186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.938517 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.938518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.939596 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.939598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.940373 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.940374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.940430 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.940433 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.940487 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.940523 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":3,"thread_start_us":116,"threads_started":1}
I20260928 06:27:30.941690 20106 tablet_service.cc:1467] Processing CreateTablet for tablet 7643de66712c448b9c142245215c29f0 (DEFAULT_TABLE table=test-table [id=7d993460c8394817b285a3d7b41442d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:30.942199 20106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7643de66712c448b9c142245215c29f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:30.942500 19975 tablet_service.cc:1467] Processing CreateTablet for tablet 7643de66712c448b9c142245215c29f0 (DEFAULT_TABLE table=test-table [id=7d993460c8394817b285a3d7b41442d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:30.942854 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7643de66712c448b9c142245215c29f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:30.945358 20317 tablet_bootstrap.cc:492] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f: Bootstrap starting.
I20260928 06:27:30.945814 20318 tablet_bootstrap.cc:492] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b: Bootstrap starting.
I20260928 06:27:30.945820 20237 tablet_service.cc:1467] Processing CreateTablet for tablet 7643de66712c448b9c142245215c29f0 (DEFAULT_TABLE table=test-table [id=7d993460c8394817b285a3d7b41442d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:27:30.946172 20317 tablet_bootstrap.cc:654] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:30.946210 20237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7643de66712c448b9c142245215c29f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:30.946540 20317 log.cc:824] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:30.946643 20318 tablet_bootstrap.cc:654] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:30.946961 20318 log.cc:824] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:30.947662 20318 tablet_bootstrap.cc:492] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b: No bootstrap required, opened a new log
I20260928 06:27:30.947763 20318 ts_tablet_manager.cc:1397] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:27:30.949476 20321 tablet_bootstrap.cc:492] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: Bootstrap starting.
I20260928 06:27:30.950150 20317 tablet_bootstrap.cc:492] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f: No bootstrap required, opened a new log
I20260928 06:27:30.950091 20318 raft_consensus.cc:348] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } }
I20260928 06:27:30.950232 20317 ts_tablet_manager.cc:1397] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 06:27:30.950256 20318 raft_consensus.cc:374] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:30.950274 20321 tablet_bootstrap.cc:654] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:30.950309 20318 raft_consensus.cc:729] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fbc4a5b3e74db7a80c061a08be553b, State: Initialized, Role: FOLLOWER
I20260928 06:27:30.950449 20318 consensus_queue.cc:260] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } }
I20260928 06:27:30.950630 20321 log.cc:824] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:30.950730 20318 ts_tablet_manager.cc:1428] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:27:30.950907 20041 heartbeater.cc:503] Master 127.31.214.126:43493 was elected leader, sending a full tablet report...
I20260928 06:27:30.951766 20321 tablet_bootstrap.cc:492] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: No bootstrap required, opened a new log
I20260928 06:27:30.951876 20321 ts_tablet_manager.cc:1397] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:27:30.953024 20317 raft_consensus.cc:348] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } }
I20260928 06:27:30.953208 20317 raft_consensus.cc:374] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:30.953248 20317 raft_consensus.cc:729] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc5434e77984c39a15a1f4f6a264b7f, State: Initialized, Role: FOLLOWER
I20260928 06:27:30.953367 20317 consensus_queue.cc:260] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } }
I20260928 06:27:30.953655 20317 ts_tablet_manager.cc:1428] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:27:30.953708 20172 heartbeater.cc:503] Master 127.31.214.126:43493 was elected leader, sending a full tablet report...
I20260928 06:27:30.954573 20321 raft_consensus.cc:348] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } }
I20260928 06:27:30.954774 20321 raft_consensus.cc:374] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:30.954813 20321 raft_consensus.cc:729] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825b4b574724490da661c3922b759abe, State: Initialized, Role: FOLLOWER
I20260928 06:27:30.954927 20321 consensus_queue.cc:260] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } }
I20260928 06:27:30.955204 20321 ts_tablet_manager.cc:1428] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:27:30.955278 20303 heartbeater.cc:503] Master 127.31.214.126:43493 was elected leader, sending a full tablet report...
I20260928 06:27:30.973732 20322 raft_consensus.cc:482] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:30.973855 20322 raft_consensus.cc:504] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } }
I20260928 06:27:30.974332 20322 leader_election.cc:288] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc5434e77984c39a15a1f4f6a264b7f (127.31.214.66:36851), 825b4b574724490da661c3922b759abe (127.31.214.67:37271)
I20260928 06:27:30.978482 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.974617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:56561 (local address 127.31.214.67:37271)
0928 06:27:30.974664 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.974667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.975018 (+ 351us) server_negotiation.cc:407] Connection header received
0928 06:27:30.975056 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.975057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.975068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.975127 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.975264 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.975951 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.977490 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.978103 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.978171 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.978180 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.978371 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.978409 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:27:30.978498 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.974522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:36851 (local address 127.31.214.65:41733)
0928 06:27:30.974605 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.974618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.974668 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.974861 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.974862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.974864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.974957 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.974960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.975964 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.975965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.977368 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.977371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.978185 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.978186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.978277 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.978284 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.978352 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.978402 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":5}
I20260928 06:27:30.978528 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.974632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.214.65:41733 (local address 127.31.214.66:36851)
0928 06:27:30.974713 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.974715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.974723 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:27:30.974754 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.974755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.974768 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.974841 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:27:30.975047 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.975879 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.977495 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.978091 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.978171 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.978180 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.978371 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.978413 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:27:30.978505 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.974537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.67:37271 (local address 127.31.214.65:56561)
0928 06:27:30.974884 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:30.974894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:30.974953 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:30.975135 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:30.975136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:30.975137 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:27:30.975194 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.975195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.976022 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.976023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.977373 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:30.977375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.978185 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.978186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:30.978277 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.978284 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:27:30.978352 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:27:30.978401 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":137,"thread_start_us":130,"threads_started":1}
I20260928 06:27:30.978904 20126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7643de66712c448b9c142245215c29f0" candidate_uuid: "99fbc4a5b3e74db7a80c061a08be553b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" is_pre_election: true
I20260928 06:27:30.979125 20126 raft_consensus.cc:2454] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fbc4a5b3e74db7a80c061a08be553b in term 0.
I20260928 06:27:30.979166 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7643de66712c448b9c142245215c29f0" candidate_uuid: "99fbc4a5b3e74db7a80c061a08be553b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825b4b574724490da661c3922b759abe" is_pre_election: true
I20260928 06:27:30.979333 20257 raft_consensus.cc:2454] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fbc4a5b3e74db7a80c061a08be553b in term 0.
I20260928 06:27:30.979375 19931 leader_election.cc:302] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc5434e77984c39a15a1f4f6a264b7f, 99fbc4a5b3e74db7a80c061a08be553b; no voters:
I20260928 06:27:30.979528 20322 raft_consensus.cc:2781] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:27:30.979590 20322 raft_consensus.cc:482] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:27:30.979614 20322 raft_consensus.cc:3037] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:30.980355 20322 raft_consensus.cc:504] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } }
I20260928 06:27:30.980515 20322 leader_election.cc:288] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [CANDIDATE]: Term 1 election: Requested vote from peers 4cc5434e77984c39a15a1f4f6a264b7f (127.31.214.66:36851), 825b4b574724490da661c3922b759abe (127.31.214.67:37271)
I20260928 06:27:30.980671 20126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7643de66712c448b9c142245215c29f0" candidate_uuid: "99fbc4a5b3e74db7a80c061a08be553b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc5434e77984c39a15a1f4f6a264b7f"
I20260928 06:27:30.980715 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7643de66712c448b9c142245215c29f0" candidate_uuid: "99fbc4a5b3e74db7a80c061a08be553b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825b4b574724490da661c3922b759abe"
I20260928 06:27:30.980769 20126 raft_consensus.cc:3037] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:30.980782 20257 raft_consensus.cc:3037] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:30.981724 20257 raft_consensus.cc:2454] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fbc4a5b3e74db7a80c061a08be553b in term 1.
I20260928 06:27:30.981724 20126 raft_consensus.cc:2454] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fbc4a5b3e74db7a80c061a08be553b in term 1.
I20260928 06:27:30.981927 19931 leader_election.cc:302] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc5434e77984c39a15a1f4f6a264b7f, 99fbc4a5b3e74db7a80c061a08be553b; no voters:
I20260928 06:27:30.982110 20322 raft_consensus.cc:2781] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:30.982306 20322 raft_consensus.cc:686] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 1 LEADER]: Becoming Leader. State: Replica: 99fbc4a5b3e74db7a80c061a08be553b, State: Running, Role: LEADER
I20260928 06:27:30.982414 20322 consensus_queue.cc:237] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } }
I20260928 06:27:30.983341 19853 catalog_manager.cc:5495] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b reported cstate change: term changed from 0 to 1, leader changed from <none> to 99fbc4a5b3e74db7a80c061a08be553b (127.31.214.65). New cstate: current_term: 1 leader_uuid: "99fbc4a5b3e74db7a80c061a08be553b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99fbc4a5b3e74db7a80c061a08be553b" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 36171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 } health_report { overall_health: UNKNOWN } } }
I20260928 06:27:30.999926 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:30.997192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.31.214.65:36171)
0928 06:27:30.997254 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:30.997256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:30.997271 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:27:30.997309 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:30.997309 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:30.997336 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:30.997400 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:30.997630 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.998245 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.999103 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:30.999296 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:30.999353 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:30.999546 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:30.999654 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:30.999718 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:30.999819 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:27:30.999840 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:27:31.002166 20126 raft_consensus.cc:1261] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [term 1 FOLLOWER]: Refusing update from remote peer 99fbc4a5b3e74db7a80c061a08be553b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:31.002183 20257 raft_consensus.cc:1261] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [term 1 FOLLOWER]: Refusing update from remote peer 99fbc4a5b3e74db7a80c061a08be553b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:31.002627 20327 consensus_queue.cc:1035] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc5434e77984c39a15a1f4f6a264b7f" member_type: VOTER last_known_addr { host: "127.31.214.66" port: 36851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:31.002772 20322 consensus_queue.cc:1035] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [LEADER]: Connected to new peer: Peer: permanent_uuid: "825b4b574724490da661c3922b759abe" member_type: VOTER last_known_addr { host: "127.31.214.67" port: 37271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:31.005335 20330 mvcc.cc:205] Tried to move back new op lower bound from 7334202781701750784 to 7334202781624745984. Current Snapshot: MvccSnapshot[applied={T|T < 7334202781701750784}]
I20260928 06:27:31.005829 20331 mvcc.cc:205] Tried to move back new op lower bound from 7334202781701750784 to 7334202781624745984. Current Snapshot: MvccSnapshot[applied={T|T < 7334202781701750784}]
I20260928 06:27:31.006137 20332 mvcc.cc:205] Tried to move back new op lower bound from 7334202781701750784 to 7334202781624745984. Current Snapshot: MvccSnapshot[applied={T|T < 7334202781701750784}]
I20260928 06:27:31.007892 19853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:31.008014 19853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:27:31.008785 19853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b73588517b964b3da5df75cc4f20feda: Sending DeleteTablet for 3 replicas of tablet 7643de66712c448b9c142245215c29f0
I20260928 06:27:31.009116 20237 tablet_service.cc:1514] Processing DeleteTablet for tablet 7643de66712c448b9c142245215c29f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:31 UTC) from {username='slave'} at 127.0.0.1:41248
I20260928 06:27:31.009160 20106 tablet_service.cc:1514] Processing DeleteTablet for tablet 7643de66712c448b9c142245215c29f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:31 UTC) from {username='slave'} at 127.0.0.1:52324
I20260928 06:27:31.009372 20340 tablet_replica.cc:330] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: stopping tablet replica
I20260928 06:27:31.009501 19975 tablet_service.cc:1514] Processing DeleteTablet for tablet 7643de66712c448b9c142245215c29f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:27:31 UTC) from {username='slave'} at 127.0.0.1:59030
I20260928 06:27:31.009724 20342 tablet_replica.cc:330] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b: stopping tablet replica
I20260928 06:27:31.009814 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 19913
I20260928 06:27:31.009848 20342 raft_consensus.cc:2229] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:27:31.010003 20342 raft_consensus.cc:2258] T 7643de66712c448b9c142245215c29f0 P 99fbc4a5b3e74db7a80c061a08be553b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:31.014146 20340 raft_consensus.cc:2229] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:31.014334 20340 raft_consensus.cc:2258] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:31.014389 20341 tablet_replica.cc:330] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f: stopping tablet replica
I20260928 06:27:31.014531 20341 raft_consensus.cc:2229] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:27:31.014670 20341 raft_consensus.cc:2258] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:27:31.015033 20340 ts_tablet_manager.cc:1905] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:27:31.015266 20341 ts_tablet_manager.cc:1905] T 7643de66712c448b9c142245215c29f0 P 4cc5434e77984c39a15a1f4f6a264b7f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:27:31.017004 19837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.214.65:36171 (error 108)
I20260928 06:27:31.017253 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 20044
I20260928 06:27:31.017319 20340 ts_tablet_manager.cc:1918] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:27:31.018335 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:31.018184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.65:36171 (local address 127.0.0.1:59056)
0928 06:27:31.018246 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:31.018303 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.65:36171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:27:31.018549 19837 catalog_manager.cc:4584] TS 99fbc4a5b3e74db7a80c061a08be553b (127.31.214.65:36171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7643de66712c448b9c142245215c29f0: Network error: Client connection negotiation failed: client connection to 127.31.214.65:36171: connect: Connection refused (error 111)
I20260928 06:27:31.018967 20340 log.cc:1196] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: Deleting WAL directory at /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790576570121568-32601-0/minicluster-data/ts-2/wal/wals/7643de66712c448b9c142245215c29f0
I20260928 06:27:31.019287 20340 ts_tablet_manager.cc:1939] T 7643de66712c448b9c142245215c29f0 P 825b4b574724490da661c3922b759abe: Deleting consensus metadata
I20260928 06:27:31.019810 19838 catalog_manager.cc:4841] TS 825b4b574724490da661c3922b759abe (127.31.214.67:37271): tablet 7643de66712c448b9c142245215c29f0 (table test-table [id=7d993460c8394817b285a3d7b41442d0]) successfully deleted
I20260928 06:27:31.023496 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 20175
I20260928 06:27:31.023520 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:31.023396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.66:36851 (local address 127.0.0.1:52340)
0928 06:27:31.023454 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:31.023482 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.66:36851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:27:31.028777 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 19822
2026-09-28T06:27:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (724 ms)
[----------] 6 tests from EncryptionPolicyTest (8706 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:31 dist-test-slave-q03f krb5kdc[20346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:31 dist-test-slave-q03f krb5kdc[20346](info): set up 2 sockets
Sep 28 06:27:31 dist-test-slave-q03f krb5kdc[20346](info): commencing operation
krb5kdc: starting...
W20260928 06:27:32.138680 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:32 dist-test-slave-q03f krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:32.273818 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:32.358213 20359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:32.358466 20359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:32.358505 20359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:32.359786 20359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:32.359861 20359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:32.359889 20359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:32.359920 20359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:32.359946 20359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:32.361501 20359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:32.361804 20359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:32.362112 20359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:32.367370 20367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:32.367400 20359 server_base.cc:1034] running on GCE node
W20260928 06:27:32.367528 20365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:32.367422 20364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:32.367835 20359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:32.368239 20359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:32.369459 20359 hybrid_clock.cc:648] HybridClock initialized: now 1790576852369450 us; error 52 us; skew 500 ppm
Sep 28 06:27:32 dist-test-slave-q03f krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:32.372437 20359 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:32.372891 20359 webserver.cc:460] Webserver started at http://127.0.0.1:43465/ using document root <none> and password file <none>
I20260928 06:27:32.373204 20359 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:32.373281 20359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:32.373437 20359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:32.374666 20359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a"
format_stamp: "Formatted at 2026-09-28 06:27:32 on dist-test-slave-q03f"
server_key: "1232484308ec8310e2a34bc81ca4b098"
server_key_iv: "c50a54a88898a2f208252d7d790fe9be"
server_key_version: "encryptionkey@0"
I20260928 06:27:32.375051 20359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a"
format_stamp: "Formatted at 2026-09-28 06:27:32 on dist-test-slave-q03f"
server_key: "1232484308ec8310e2a34bc81ca4b098"
server_key_iv: "c50a54a88898a2f208252d7d790fe9be"
server_key_version: "encryptionkey@0"
I20260928 06:27:32.376441 20359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:27:32.377266 20374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:32.377493 20359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:32.377626 20359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a"
format_stamp: "Formatted at 2026-09-28 06:27:32 on dist-test-slave-q03f"
server_key: "1232484308ec8310e2a34bc81ca4b098"
server_key_iv: "c50a54a88898a2f208252d7d790fe9be"
server_key_version: "encryptionkey@0"
I20260928 06:27:32.377707 20359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:32.385805 20359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:32.388043 20359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:32.388307 20359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:32.392743 20359 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39901
I20260928 06:27:32.392776 20426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39901 every 8 connection(s)
I20260928 06:27:32.393144 20359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:32.394163 20427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:32.396473 20427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a: Bootstrap starting.
I20260928 06:27:32.397025 20427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:32.397264 20427 log.cc:824] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:32.398074 20427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a: No bootstrap required, opened a new log
I20260928 06:27:32.399361 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 20359
I20260928 06:27:32.399456 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:32.399655 32601 external_mini_cluster.cc:1428] Setting key 3818626922c6a93ac88961e2368e9ab2
I20260928 06:27:32.399852 20427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39901 } }
I20260928 06:27:32.400022 20427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:32.400068 20427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef40fa2cfa142a7be1b5824d98c7a4a, State: Initialized, Role: FOLLOWER
I20260928 06:27:32.400235 20427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39901 } }
I20260928 06:27:32.400336 20427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:32.400374 20427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:32.400429 20427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:32.401680 20427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39901 } }
I20260928 06:27:32.401835 20427 leader_election.cc:302] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ef40fa2cfa142a7be1b5824d98c7a4a; no voters:
I20260928 06:27:32.402027 20427 leader_election.cc:288] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:32.402058 20432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:32.402169 20432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [term 1 LEADER]: Becoming Leader. State: Replica: 2ef40fa2cfa142a7be1b5824d98c7a4a, State: Running, Role: LEADER
I20260928 06:27:32.402256 20432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39901 } }
I20260928 06:27:32.402323 20427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:32.402583 20434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39901 } } }
I20260928 06:27:32.402599 20433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ef40fa2cfa142a7be1b5824d98c7a4a. Latest consensus state: current_term: 1 leader_uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef40fa2cfa142a7be1b5824d98c7a4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39901 } } }
I20260928 06:27:32.402765 20434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:32.402786 20433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:32.403736 20440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:32.404315 20440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:32.405857 20440 catalog_manager.cc:1347] Generated new cluster ID: ea4c5f01e1c241759b842c4439e6cfa8
I20260928 06:27:32.405915 20440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:27:32 dist-test-slave-q03f krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:27:32.411994 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:32.400249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36859 (local address 127.0.0.1:39901)
0928 06:27:32.400468 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:32.400479 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:32.400600 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:27:32.400731 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:32.400736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:32.400789 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:32.400903 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:32.401833 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:32.403274 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:32.404440 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:32.405051 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:32.405162 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:32.408054 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:32.408058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:32.408059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:32.408071 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:32.410191 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:32.410625 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:32.410629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:32.410629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:32.410748 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:32.410989 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:32.410991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:32.410991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:32.411387 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:32.411591 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:32.411675 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:27:32.411791 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260928 06:27:32.413043 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:32.416743 20440 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:32.417407 20440 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:32.421448 20440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ef40fa2cfa142a7be1b5824d98c7a4a: Generated new TSK 0
I20260928 06:27:32.421628 20440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:32.422132 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:32.414591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48799 (local address 127.0.0.1:39901)
0928 06:27:32.414622 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:32.414624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:32.414801 (+ 177us) server_negotiation.cc:407] Connection header received
0928 06:27:32.414854 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:32.414854 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:32.414864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:32.414934 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:32.415574 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:32.416255 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:32.417289 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:32.417609 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:32.417682 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:32.419073 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:32.419075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:32.419075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:32.419082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:32.420739 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:32.421116 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:32.421117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:32.421117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:32.421176 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:32.421426 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:32.421427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:32.421427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:32.421542 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:32.421617 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:32.422011 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 06:27:32.422076 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:27:32.423919 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 20359
W20260928 06:27:32.429658 20344 connection.cc:535] client connection to 127.0.0.1:39901 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39901): Connection reset by peer (error 104)
2026-09-28T06:27:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:32 dist-test-slave-q03f krb5kdc[20458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:32 dist-test-slave-q03f krb5kdc[20458](info): set up 2 sockets
Sep 28 06:27:32 dist-test-slave-q03f krb5kdc[20458](info): commencing operation
krb5kdc: starting...
W20260928 06:27:33.498189 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:33 dist-test-slave-q03f krb5kdc[20458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:33.625069 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:33.708244 20471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:33.708590 20471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:33.708632 20471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:33.709816 20471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:33.709892 20471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:33.709916 20471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:33.709939 20471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:33.709956 20471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:33.711524 20471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:33.711827 20471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:33.712080 20471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:27:33.717465 20471 server_base.cc:1034] running on GCE node
W20260928 06:27:33.717470 20476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:33.717444 20477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:33.717442 20479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:33.718046 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:33.718328 20471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:33.719504 20471 hybrid_clock.cc:648] HybridClock initialized: now 1790576853719481 us; error 39 us; skew 500 ppm
Sep 28 06:27:33 dist-test-slave-q03f krb5kdc[20458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:33.722323 20471 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:33.722846 20471 webserver.cc:460] Webserver started at http://127.0.0.1:35745/ using document root <none> and password file <none>
I20260928 06:27:33.723065 20471 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:33.723110 20471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:33.723202 20471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:33.724251 20471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "ca66e92cd4644bd4aaebffca3804fef6"
format_stamp: "Formatted at 2026-09-28 06:27:33 on dist-test-slave-q03f"
server_key: "ee668a5c781b897912c9d0b9dbbda003"
server_key_iv: "d195e27a3b8b489d063a71affc4c844b"
server_key_version: "encryptionkey@0"
I20260928 06:27:33.724568 20471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "ca66e92cd4644bd4aaebffca3804fef6"
format_stamp: "Formatted at 2026-09-28 06:27:33 on dist-test-slave-q03f"
server_key: "ee668a5c781b897912c9d0b9dbbda003"
server_key_iv: "d195e27a3b8b489d063a71affc4c844b"
server_key_version: "encryptionkey@0"
I20260928 06:27:33.725893 20471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:27:33.726698 20486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:33.726866 20471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:27:33.726959 20471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "ca66e92cd4644bd4aaebffca3804fef6"
format_stamp: "Formatted at 2026-09-28 06:27:33 on dist-test-slave-q03f"
server_key: "ee668a5c781b897912c9d0b9dbbda003"
server_key_iv: "d195e27a3b8b489d063a71affc4c844b"
server_key_version: "encryptionkey@0"
I20260928 06:27:33.727017 20471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:33.743492 20471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:33.745586 20471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:33.745703 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:33.749725 20471 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45833
I20260928 06:27:33.749811 20538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45833 every 8 connection(s)
I20260928 06:27:33.750187 20471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:33.750507 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 20471
I20260928 06:27:33.750651 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:33.750908 32601 external_mini_cluster.cc:1428] Setting key c44ca0765231a35338e3fa93f1978a29
I20260928 06:27:33.751400 20539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:33.754892 20539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6: Bootstrap starting.
I20260928 06:27:33.755630 20539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:33.755975 20539 log.cc:824] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:33.756845 20539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6: No bootstrap required, opened a new log
I20260928 06:27:33.757289 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:33.751476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49295 (local address 127.0.0.1:45833)
0928 06:27:33.751698 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:33.751708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:33.751740 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:27:33.752265 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:33.752274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:33.752341 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:33.752509 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:33.752715 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:33.754138 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:33.754915 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:33.755528 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:33.755656 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:33.756459 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:33.756519 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:33.756519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:33.756744 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:33.756789 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:33.756805 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:33.756896 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:33.757033 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:27:33.757062 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":121,"threads_started":1}
I20260928 06:27:33.759378 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:33.759373 20539 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca66e92cd4644bd4aaebffca3804fef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45833 } }
I20260928 06:27:33.759562 20539 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:33.759606 20539 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca66e92cd4644bd4aaebffca3804fef6, State: Initialized, Role: FOLLOWER
I20260928 06:27:33.759789 20539 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca66e92cd4644bd4aaebffca3804fef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45833 } }
I20260928 06:27:33.759891 20539 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:33.759922 20539 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:33.759967 20539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:33.760721 20539 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca66e92cd4644bd4aaebffca3804fef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45833 } }
I20260928 06:27:33.760841 20539 leader_election.cc:302] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca66e92cd4644bd4aaebffca3804fef6; no voters:
I20260928 06:27:33.760916 20544 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:27:33.760972 20544 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca66e92cd4644bd4aaebffca3804fef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45833 } }
I20260928 06:27:33.761029 20539 leader_election.cc:288] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:33.761070 20544 leader_election.cc:302] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca66e92cd4644bd4aaebffca3804fef6; no voters:
I20260928 06:27:33.761188 20544 leader_election.cc:288] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260928 06:27:33.761246 20551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:33.761277 20539 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:33.761315 20551 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [term 1 LEADER]: Becoming Leader. State: Replica: ca66e92cd4644bd4aaebffca3804fef6, State: Running, Role: LEADER
I20260928 06:27:33.761404 20551 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca66e92cd4644bd4aaebffca3804fef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45833 } }
I20260928 06:27:33.761792 20544 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca66e92cd4644bd4aaebffca3804fef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca66e92cd4644bd4aaebffca3804fef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45833 } } }
I20260928 06:27:33.761806 20551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca66e92cd4644bd4aaebffca3804fef6. Latest consensus state: current_term: 1 leader_uuid: "ca66e92cd4644bd4aaebffca3804fef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca66e92cd4644bd4aaebffca3804fef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45833 } } }
I20260928 06:27:33.761914 20551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:33.761909 20544 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:33.762203 20558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:33.762773 20558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:33.766705 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:33.762985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54261 (local address 127.0.0.1:45833)
0928 06:27:33.763052 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:33.763054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:33.763335 (+ 281us) server_negotiation.cc:407] Connection header received
0928 06:27:33.763364 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:33.763364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:33.763377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:33.763478 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:33.763693 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:33.764264 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:33.765169 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:33.765431 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:33.765532 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:33.766421 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:33.766422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:33.766423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:33.766427 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:33.766445 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:33.766449 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:33.766528 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:33.766612 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:27:33.766626 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:27:33.767474 20558 catalog_manager.cc:1347] Generated new cluster ID: cd827b4e13f441a3a985012facd551ae
I20260928 06:27:33.767544 20558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:33.789016 20558 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:33.789572 20558 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:33.798507 20558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca66e92cd4644bd4aaebffca3804fef6: Generated new TSK 0
I20260928 06:27:33.798748 20558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:33.809402 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 20471
W20260928 06:27:33.814219 20456 connection.cc:535] client connection to 127.0.0.1:45833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45833): Connection reset by peer (error 104)
2026-09-28T06:27:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:33 dist-test-slave-q03f krb5kdc[20571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:33 dist-test-slave-q03f krb5kdc[20571](info): set up 2 sockets
Sep 28 06:27:33 dist-test-slave-q03f krb5kdc[20571](info): commencing operation
krb5kdc: starting...
W20260928 06:27:34.870117 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:34 dist-test-slave-q03f krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:34.998230 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:35.086275 20584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:35.086572 20584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:35.086617 20584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:35.088099 20584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:35.088207 20584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:35.088235 20584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:35.088255 20584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:35.088315 20584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:35.090054 20584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:35.090377 20584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:35.090627 20584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:35.096047 20589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:35.096055 20592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:35.096128 20590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:35.096156 20584 server_base.cc:1034] running on GCE node
I20260928 06:27:35.096468 20584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:35.096741 20584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:35.097927 20584 hybrid_clock.cc:648] HybridClock initialized: now 1790576855097906 us; error 33 us; skew 500 ppm
Sep 28 06:27:35 dist-test-slave-q03f krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:35.100602 20584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:35.101164 20584 webserver.cc:460] Webserver started at http://127.0.0.1:35183/ using document root <none> and password file <none>
I20260928 06:27:35.101403 20584 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:35.101473 20584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:35.101630 20584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:35.102826 20584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "602b1ea4da0b4d0b9681c4ba14df10cd"
format_stamp: "Formatted at 2026-09-28 06:27:35 on dist-test-slave-q03f"
server_key: "e6a0cdfbb3846f582b1e08ec211adee7"
server_key_iv: "fcd9c67dec934ebd2f31bca069e97be2"
server_key_version: "encryptionkey@0"
I20260928 06:27:35.103204 20584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "602b1ea4da0b4d0b9681c4ba14df10cd"
format_stamp: "Formatted at 2026-09-28 06:27:35 on dist-test-slave-q03f"
server_key: "e6a0cdfbb3846f582b1e08ec211adee7"
server_key_iv: "fcd9c67dec934ebd2f31bca069e97be2"
server_key_version: "encryptionkey@0"
I20260928 06:27:35.104621 20584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:27:35.105466 20599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:35.105733 20584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:35.105916 20584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "602b1ea4da0b4d0b9681c4ba14df10cd"
format_stamp: "Formatted at 2026-09-28 06:27:35 on dist-test-slave-q03f"
server_key: "e6a0cdfbb3846f582b1e08ec211adee7"
server_key_iv: "fcd9c67dec934ebd2f31bca069e97be2"
server_key_version: "encryptionkey@0"
I20260928 06:27:35.106053 20584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:35.109594 20584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:35.111919 20584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:35.112102 20584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:35.116334 20584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40011
I20260928 06:27:35.116369 20651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40011 every 8 connection(s)
I20260928 06:27:35.116704 20584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:35.117748 20652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:35.120139 20652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd: Bootstrap starting.
I20260928 06:27:35.120712 20652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:35.120944 20652 log.cc:824] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:35.121599 20652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd: No bootstrap required, opened a new log
I20260928 06:27:35.123250 20652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602b1ea4da0b4d0b9681c4ba14df10cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40011 } }
I20260928 06:27:35.123468 20652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:35.123512 20652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602b1ea4da0b4d0b9681c4ba14df10cd, State: Initialized, Role: FOLLOWER
I20260928 06:27:35.123571 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 20584
I20260928 06:27:35.123694 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:35.123687 20652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602b1ea4da0b4d0b9681c4ba14df10cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40011 } }
I20260928 06:27:35.123780 20652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:35.123814 20652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:35.123860 20652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:35.123916 32601 external_mini_cluster.cc:1428] Setting key cc8ae7d199ae4572013422c60b30f4cd
I20260928 06:27:35.124626 20652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602b1ea4da0b4d0b9681c4ba14df10cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40011 } }
I20260928 06:27:35.124747 20652 leader_election.cc:302] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 602b1ea4da0b4d0b9681c4ba14df10cd; no voters:
I20260928 06:27:35.124936 20652 leader_election.cc:288] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:35.124972 20657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:35.125079 20657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [term 1 LEADER]: Becoming Leader. State: Replica: 602b1ea4da0b4d0b9681c4ba14df10cd, State: Running, Role: LEADER
I20260928 06:27:35.125169 20657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602b1ea4da0b4d0b9681c4ba14df10cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40011 } }
I20260928 06:27:35.125209 20652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:35.125479 20657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "602b1ea4da0b4d0b9681c4ba14df10cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602b1ea4da0b4d0b9681c4ba14df10cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40011 } } }
I20260928 06:27:35.125557 20657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:35.125504 20659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 602b1ea4da0b4d0b9681c4ba14df10cd. Latest consensus state: current_term: 1 leader_uuid: "602b1ea4da0b4d0b9681c4ba14df10cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "602b1ea4da0b4d0b9681c4ba14df10cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40011 } } }
I20260928 06:27:35.125646 20659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:35.125777 20661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:35.126366 20661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:35.128167 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:35.124480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48127 (local address 127.0.0.1:40011)
0928 06:27:35.124711 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:35.124722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:35.124760 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:27:35.124905 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:35.124917 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:35.124954 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:35.125128 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:35.127674 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:35.127678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:35.127678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:35.127683 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:35.127737 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:35.127769 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:35.127848 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:35.127933 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:27:35.127975 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":126,"threads_started":1}
I20260928 06:27:35.128894 20661 catalog_manager.cc:1347] Generated new cluster ID: d5c3869a132a438c85270b5da2913b9f
I20260928 06:27:35.128948 20661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:35.129127 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:35.131973 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:35.130869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41759 (local address 127.0.0.1:40011)
0928 06:27:35.130924 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:35.130925 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:35.131012 (+ 87us) server_negotiation.cc:407] Connection header received
0928 06:27:35.131076 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:35.131077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:35.131082 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:35.131128 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:35.131739 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:35.131740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:35.131740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:35.131744 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:35.131765 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:35.131769 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:35.131824 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:35.131900 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:27:35.131913 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:27:35.141855 20661 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:35.142103 20661 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:35.151130 20661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 602b1ea4da0b4d0b9681c4ba14df10cd: Generated new TSK 0
I20260928 06:27:35.151376 20661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:35.156260 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 20584
2026-09-28T06:27:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:35 dist-test-slave-q03f krb5kdc[20683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:35 dist-test-slave-q03f krb5kdc[20683](info): set up 2 sockets
Sep 28 06:27:35 dist-test-slave-q03f krb5kdc[20683](info): commencing operation
krb5kdc: starting...
W20260928 06:27:36.210321 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:36 dist-test-slave-q03f krb5kdc[20683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:36.337644 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:36.421793 20696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:36.422044 20696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:36.422071 20696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:36.423295 20696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:36.423352 20696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:36.423365 20696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:36.423378 20696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:36.423390 20696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:36.424894 20696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:36.425119 20696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:36.425307 20696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:36.430594 20702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:36.430593 20704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:36.430583 20701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:36.430931 20696 server_base.cc:1034] running on GCE node
I20260928 06:27:36.431191 20696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:36.431507 20696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:36.432677 20696 hybrid_clock.cc:648] HybridClock initialized: now 1790576856432663 us; error 48 us; skew 500 ppm
Sep 28 06:27:36 dist-test-slave-q03f krb5kdc[20683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:36.435585 20696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:36.436146 20696 webserver.cc:460] Webserver started at http://127.0.0.1:45097/ using document root <none> and password file <none>
I20260928 06:27:36.436406 20696 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:36.436482 20696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:36.436647 20696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:36.437862 20696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "b941d9dc2a8a49fdb960ca39afacb669"
format_stamp: "Formatted at 2026-09-28 06:27:36 on dist-test-slave-q03f"
server_key: "6df65d4233da03ee2a8e839d07a9e6be"
server_key_iv: "5b014749605d881d256d7261e0251491"
server_key_version: "encryptionkey@0"
I20260928 06:27:36.438320 20696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "b941d9dc2a8a49fdb960ca39afacb669"
format_stamp: "Formatted at 2026-09-28 06:27:36 on dist-test-slave-q03f"
server_key: "6df65d4233da03ee2a8e839d07a9e6be"
server_key_iv: "5b014749605d881d256d7261e0251491"
server_key_version: "encryptionkey@0"
I20260928 06:27:36.439879 20696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:27:36.440752 20711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:36.441031 20696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:36.441148 20696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "b941d9dc2a8a49fdb960ca39afacb669"
format_stamp: "Formatted at 2026-09-28 06:27:36 on dist-test-slave-q03f"
server_key: "6df65d4233da03ee2a8e839d07a9e6be"
server_key_iv: "5b014749605d881d256d7261e0251491"
server_key_version: "encryptionkey@0"
I20260928 06:27:36.441247 20696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:36.456110 20696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:36.459231 20696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:36.459511 20696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:36.464238 20696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33829
I20260928 06:27:36.464272 20763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33829 every 8 connection(s)
I20260928 06:27:36.464697 20696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:36.465728 20764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:36.468176 20764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669: Bootstrap starting.
I20260928 06:27:36.468804 20764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:36.469115 20764 log.cc:824] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:36.469915 20764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669: No bootstrap required, opened a new log
I20260928 06:27:36.471626 20764 raft_consensus.cc:348] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b941d9dc2a8a49fdb960ca39afacb669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260928 06:27:36.471774 20764 raft_consensus.cc:374] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:36.471797 20764 raft_consensus.cc:729] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b941d9dc2a8a49fdb960ca39afacb669, State: Initialized, Role: FOLLOWER
I20260928 06:27:36.471946 20764 consensus_queue.cc:260] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b941d9dc2a8a49fdb960ca39afacb669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260928 06:27:36.472031 20764 raft_consensus.cc:388] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:36.472056 20764 raft_consensus.cc:482] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:36.472085 20764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:36.472616 20764 raft_consensus.cc:504] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b941d9dc2a8a49fdb960ca39afacb669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260928 06:27:36.472704 20764 leader_election.cc:302] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b941d9dc2a8a49fdb960ca39afacb669; no voters:
I20260928 06:27:36.472863 20764 leader_election.cc:288] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:36.472923 20767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:36.473052 20767 raft_consensus.cc:686] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [term 1 LEADER]: Becoming Leader. State: Replica: b941d9dc2a8a49fdb960ca39afacb669, State: Running, Role: LEADER
I20260928 06:27:36.473106 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 20696
I20260928 06:27:36.473157 20764 sys_catalog.cc:564] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:36.473133 20767 consensus_queue.cc:237] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b941d9dc2a8a49fdb960ca39afacb669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260928 06:27:36.473225 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:36.473492 32601 external_mini_cluster.cc:1428] Setting key 47dc776819f029c400a4a9b72d83cc94
I20260928 06:27:36.473487 20767 sys_catalog.cc:455] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b941d9dc2a8a49fdb960ca39afacb669. Latest consensus state: current_term: 1 leader_uuid: "b941d9dc2a8a49fdb960ca39afacb669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b941d9dc2a8a49fdb960ca39afacb669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } } }
I20260928 06:27:36.473562 20767 sys_catalog.cc:458] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:36.473732 20768 sys_catalog.cc:455] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b941d9dc2a8a49fdb960ca39afacb669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b941d9dc2a8a49fdb960ca39afacb669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } } }
I20260928 06:27:36.473872 20768 sys_catalog.cc:458] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:36.474046 20771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:36.474861 20771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:36.476598 20771 catalog_manager.cc:1347] Generated new cluster ID: 08cc2550035c4a658d40fab40e5f0856
I20260928 06:27:36.476657 20771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:27:36 dist-test-slave-q03f krb5kdc[20683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:27:36.485131 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:36.474257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59107 (local address 127.0.0.1:33829)
0928 06:27:36.474480 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:36.474492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:36.474536 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:27:36.475001 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:36.475018 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:36.475197 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:36.476083 (+ 886us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:36.476198 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:36.477255 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:36.478135 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:36.478563 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:36.478688 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:36.481125 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:36.481129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:36.481130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:36.481149 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:36.483509 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:36.483905 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:36.483908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:36.483909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:36.484021 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:36.484248 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:36.484249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:36.484250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:36.484514 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:36.484697 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:36.484782 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:27:36.484937 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
I20260928 06:27:36.486160 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:36.492221 20771 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:36.492964 20771 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:36.497051 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:36.487732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60911 (local address 127.0.0.1:33829)
0928 06:27:36.487787 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:36.487789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:36.490195 (+ 2406us) server_negotiation.cc:407] Connection header received
0928 06:27:36.490290 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:36.490291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:36.490304 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:36.490359 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:36.491039 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:36.491575 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:36.492566 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:36.492796 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:36.492876 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:36.494282 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:36.494283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:36.494284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:36.494292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:36.495846 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:36.496215 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:36.496217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:36.496217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:36.496283 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:36.496489 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:36.496490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:36.496490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:36.496610 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:36.496730 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:36.496847 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 06:27:36.496943 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:27:36.502099 20771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b941d9dc2a8a49fdb960ca39afacb669: Generated new TSK 0
I20260928 06:27:36.502324 20771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:36.505308 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 20696
2026-09-28T06:27:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:36 dist-test-slave-q03f krb5kdc[20795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:36 dist-test-slave-q03f krb5kdc[20795](info): set up 2 sockets
Sep 28 06:27:36 dist-test-slave-q03f krb5kdc[20795](info): commencing operation
krb5kdc: starting...
W20260928 06:27:37.574340 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:37 dist-test-slave-q03f krb5kdc[20795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:37.705346 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:37.791208 20808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:37.791491 20808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:37.791534 20808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:37.792773 20808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:37.792846 20808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:37.792873 20808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:37.792905 20808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:37.792939 20808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:37.794739 20808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:37.795044 20808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:37.795346 20808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:37.800981 20813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:37.800978 20816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:37.800981 20814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:37.801340 20808 server_base.cc:1034] running on GCE node
I20260928 06:27:37.801663 20808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:37.802131 20808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:37.803305 20808 hybrid_clock.cc:648] HybridClock initialized: now 1790576857803293 us; error 30 us; skew 500 ppm
Sep 28 06:27:37 dist-test-slave-q03f krb5kdc[20795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:37.810642 20808 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:37.811178 20808 webserver.cc:460] Webserver started at http://127.0.0.1:34653/ using document root <none> and password file <none>
I20260928 06:27:37.811456 20808 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:37.811530 20808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:37.811689 20808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:37.812983 20808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "f8094ebd685c479c8035ea5dad17bf1f"
format_stamp: "Formatted at 2026-09-28 06:27:37 on dist-test-slave-q03f"
server_key: "8d1d16a0e5f6e6560a0662bfb2ea5247"
server_key_iv: "2b2c08bfa821eb001af50b8835cb1d3d"
server_key_version: "encryptionkey@0"
I20260928 06:27:37.813357 20808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "f8094ebd685c479c8035ea5dad17bf1f"
format_stamp: "Formatted at 2026-09-28 06:27:37 on dist-test-slave-q03f"
server_key: "8d1d16a0e5f6e6560a0662bfb2ea5247"
server_key_iv: "2b2c08bfa821eb001af50b8835cb1d3d"
server_key_version: "encryptionkey@0"
I20260928 06:27:37.814936 20808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:27:37.815717 20823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:37.815939 20808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:37.816051 20808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "f8094ebd685c479c8035ea5dad17bf1f"
format_stamp: "Formatted at 2026-09-28 06:27:37 on dist-test-slave-q03f"
server_key: "8d1d16a0e5f6e6560a0662bfb2ea5247"
server_key_iv: "2b2c08bfa821eb001af50b8835cb1d3d"
server_key_version: "encryptionkey@0"
I20260928 06:27:37.816128 20808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:37.826426 20808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:37.828953 20808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:37.829128 20808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:37.833603 20808 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40651
I20260928 06:27:37.833647 20875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40651 every 8 connection(s)
I20260928 06:27:37.834146 20808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:37.835189 20876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:37.837476 20876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f: Bootstrap starting.
I20260928 06:27:37.838160 20876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:37.838407 20876 log.cc:824] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:37.839134 20876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f: No bootstrap required, opened a new log
I20260928 06:27:37.840720 20876 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8094ebd685c479c8035ea5dad17bf1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40651 } }
I20260928 06:27:37.840873 20876 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:37.840896 20876 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8094ebd685c479c8035ea5dad17bf1f, State: Initialized, Role: FOLLOWER
I20260928 06:27:37.841040 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 20808
I20260928 06:27:37.841068 20876 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8094ebd685c479c8035ea5dad17bf1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40651 } }
I20260928 06:27:37.841171 20876 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:37.841183 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:37.841238 20876 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:37.841296 20876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:37.841404 32601 external_mini_cluster.cc:1428] Setting key a7373c8acfdccc7c202c489598c0786d
I20260928 06:27:37.842064 20876 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8094ebd685c479c8035ea5dad17bf1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40651 } }
I20260928 06:27:37.842195 20876 leader_election.cc:302] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8094ebd685c479c8035ea5dad17bf1f; no voters:
I20260928 06:27:37.842388 20876 leader_election.cc:288] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:37.842420 20881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:37.842607 20881 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [term 1 LEADER]: Becoming Leader. State: Replica: f8094ebd685c479c8035ea5dad17bf1f, State: Running, Role: LEADER
I20260928 06:27:37.842674 20876 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:37.842725 20881 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8094ebd685c479c8035ea5dad17bf1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40651 } }
I20260928 06:27:37.843362 20881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8094ebd685c479c8035ea5dad17bf1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8094ebd685c479c8035ea5dad17bf1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40651 } } }
I20260928 06:27:37.843508 20881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:37.843638 20881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8094ebd685c479c8035ea5dad17bf1f. Latest consensus state: current_term: 1 leader_uuid: "f8094ebd685c479c8035ea5dad17bf1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8094ebd685c479c8035ea5dad17bf1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40651 } } }
I20260928 06:27:37.843706 20890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:37.843781 20881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:37.844327 20890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:37.846189 20890 catalog_manager.cc:1347] Generated new cluster ID: 4c424976340b426ea7250754ae7c07ed
I20260928 06:27:37.846295 20890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:37.847748 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:37.841967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54893 (local address 127.0.0.1:40651)
0928 06:27:37.842135 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:37.842163 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:37.842199 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:27:37.842343 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:37.842349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:37.842395 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:37.842549 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:37.842801 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:37.844227 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:37.845437 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:37.846053 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:37.846192 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:37.846990 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:37.846994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:37.846995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:37.847200 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:37.847231 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:37.847242 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:37.847328 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:37.847406 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:27:37.847578 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260928 06:27:37.848723 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:37.853399 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:37.850290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33679 (local address 127.0.0.1:40651)
0928 06:27:37.850329 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:37.850331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:37.850504 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:27:37.850571 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:37.850572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:37.850583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:37.850638 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:37.850902 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:37.851406 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:37.852158 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:37.852381 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:37.852458 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:37.853156 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:37.853158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:37.853158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:37.853164 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:37.853178 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:37.853183 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:37.853231 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:37.853288 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:27:37.853314 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:27:37.859952 20890 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:37.860585 20890 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:37.868347 20890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8094ebd685c479c8035ea5dad17bf1f: Generated new TSK 0
I20260928 06:27:37.868628 20890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:37.871867 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 20808
2026-09-28T06:27:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:37 dist-test-slave-q03f krb5kdc[20907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:37 dist-test-slave-q03f krb5kdc[20907](info): set up 2 sockets
Sep 28 06:27:37 dist-test-slave-q03f krb5kdc[20907](info): commencing operation
krb5kdc: starting...
W20260928 06:27:38.918556 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:39 dist-test-slave-q03f krb5kdc[20907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:39.050359 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:39.134232 20920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:39.134462 20920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:39.134487 20920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:39.135617 20920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:39.135654 20920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:39.135668 20920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:39.135681 20920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:39.135694 20920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:39.137095 20920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:39.137295 20920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:39.137467 20920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:39.142750 20928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:39.142773 20925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:39.142874 20926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:39.143251 20920 server_base.cc:1034] running on GCE node
I20260928 06:27:39.143581 20920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:39.143920 20920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:39.145098 20920 hybrid_clock.cc:648] HybridClock initialized: now 1790576859145084 us; error 34 us; skew 500 ppm
Sep 28 06:27:39 dist-test-slave-q03f krb5kdc[20907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:39.147857 20920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:39.148404 20920 webserver.cc:460] Webserver started at http://127.0.0.1:36331/ using document root <none> and password file <none>
I20260928 06:27:39.148648 20920 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:39.148720 20920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:39.148882 20920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:39.150180 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "e245ab092f514ae58306abf9cb627a7b"
format_stamp: "Formatted at 2026-09-28 06:27:39 on dist-test-slave-q03f"
server_key: "2b8d0c598c95aebbe3d11e61d7941ef8"
server_key_iv: "81e3f167cedc6d03ca1aab69e1fb9b01"
server_key_version: "encryptionkey@0"
I20260928 06:27:39.150583 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "e245ab092f514ae58306abf9cb627a7b"
format_stamp: "Formatted at 2026-09-28 06:27:39 on dist-test-slave-q03f"
server_key: "2b8d0c598c95aebbe3d11e61d7941ef8"
server_key_iv: "81e3f167cedc6d03ca1aab69e1fb9b01"
server_key_version: "encryptionkey@0"
I20260928 06:27:39.152082 20920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:27:39.153175 20935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:39.153517 20920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:39.153637 20920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "e245ab092f514ae58306abf9cb627a7b"
format_stamp: "Formatted at 2026-09-28 06:27:39 on dist-test-slave-q03f"
server_key: "2b8d0c598c95aebbe3d11e61d7941ef8"
server_key_iv: "81e3f167cedc6d03ca1aab69e1fb9b01"
server_key_version: "encryptionkey@0"
I20260928 06:27:39.153721 20920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:39.157442 20920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:39.160086 20920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:39.160291 20920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:39.164628 20920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45087
I20260928 06:27:39.164706 20987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45087 every 8 connection(s)
I20260928 06:27:39.165112 20920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:39.165439 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 20920
I20260928 06:27:39.165560 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:39.165817 32601 external_mini_cluster.cc:1428] Setting key 01a72673a6bf8491c9fb344bfdbe34d2
I20260928 06:27:39.166478 20988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:39.168582 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:39.166558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50185 (local address 127.0.0.1:45087)
0928 06:27:39.166758 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:39.166774 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:39.166801 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:27:39.167226 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:39.167232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:39.167257 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:39.167387 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:39.168116 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:39.168120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:39.168120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:39.168124 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:39.168164 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:39.168175 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:39.168233 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:39.168321 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:27:39.168384 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
I20260928 06:27:39.169631 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:39.170114 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b: Bootstrap starting.
I20260928 06:27:39.170725 20988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:39.171026 20988 log.cc:824] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:39.171835 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b: No bootstrap required, opened a new log
I20260928 06:27:39.172530 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:39.171171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44113 (local address 127.0.0.1:45087)
0928 06:27:39.171234 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:39.171236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:39.171350 (+ 114us) server_negotiation.cc:407] Connection header received
0928 06:27:39.171375 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:39.171376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:39.171381 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:39.171449 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:39.172258 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:39.172259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:39.172260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:39.172264 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:39.172281 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:39.172291 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:39.172361 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:39.172429 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:27:39.172445 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:27:39.173920 20988 raft_consensus.cc:348] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245ab092f514ae58306abf9cb627a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260928 06:27:39.174171 20988 raft_consensus.cc:374] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:39.174201 20988 raft_consensus.cc:729] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e245ab092f514ae58306abf9cb627a7b, State: Initialized, Role: FOLLOWER
I20260928 06:27:39.174317 20988 consensus_queue.cc:260] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245ab092f514ae58306abf9cb627a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260928 06:27:39.174396 20988 raft_consensus.cc:388] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:39.174417 20988 raft_consensus.cc:482] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:39.174477 20988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:39.175034 20988 raft_consensus.cc:504] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245ab092f514ae58306abf9cb627a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260928 06:27:39.175120 20988 leader_election.cc:302] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e245ab092f514ae58306abf9cb627a7b; no voters:
I20260928 06:27:39.175251 20988 leader_election.cc:288] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:39.175366 20998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:39.175518 20998 raft_consensus.cc:686] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [term 1 LEADER]: Becoming Leader. State: Replica: e245ab092f514ae58306abf9cb627a7b, State: Running, Role: LEADER
I20260928 06:27:39.175536 20988 sys_catalog.cc:564] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:39.175613 20998 consensus_queue.cc:237] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245ab092f514ae58306abf9cb627a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } }
I20260928 06:27:39.176003 21000 sys_catalog.cc:455] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e245ab092f514ae58306abf9cb627a7b. Latest consensus state: current_term: 1 leader_uuid: "e245ab092f514ae58306abf9cb627a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245ab092f514ae58306abf9cb627a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } } }
I20260928 06:27:39.176105 21000 sys_catalog.cc:458] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:39.175992 20999 sys_catalog.cc:455] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e245ab092f514ae58306abf9cb627a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e245ab092f514ae58306abf9cb627a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45087 } } }
I20260928 06:27:39.176302 21005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:39.176605 20999 sys_catalog.cc:458] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:39.177090 21005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:39.178684 21005 catalog_manager.cc:1347] Generated new cluster ID: d5ab876c0be8433abd840817bdfe0808
I20260928 06:27:39.178738 21005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:39.183331 21005 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:39.183503 21005 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:39.188990 21005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e245ab092f514ae58306abf9cb627a7b: Generated new TSK 0
I20260928 06:27:39.189189 21005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:39.192765 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 20920
2026-09-28T06:27:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:39 dist-test-slave-q03f krb5kdc[21019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:39 dist-test-slave-q03f krb5kdc[21019](info): set up 2 sockets
Sep 28 06:27:39 dist-test-slave-q03f krb5kdc[21019](info): commencing operation
krb5kdc: starting...
W20260928 06:27:40.238250 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:40 dist-test-slave-q03f krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:40.366583 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:40.451450 21032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:40.451725 21032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:40.451768 21032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:40.452965 21032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:40.453040 21032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:40.453065 21032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:40.453094 21032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:40.453125 21032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:40.454687 21032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:40.455044 21032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:40.455310 21032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:40.460853 21038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:40.460762 21040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:40.460595 21037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:40.461089 21032 server_base.cc:1034] running on GCE node
I20260928 06:27:40.461462 21032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:40.461884 21032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:40.463119 21032 hybrid_clock.cc:648] HybridClock initialized: now 1790576860463091 us; error 46 us; skew 500 ppm
Sep 28 06:27:40 dist-test-slave-q03f krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:40.465932 21032 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:40.466459 21032 webserver.cc:460] Webserver started at http://127.0.0.1:42659/ using document root <none> and password file <none>
I20260928 06:27:40.466696 21032 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:40.466766 21032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:40.466924 21032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:40.468053 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "6597e5ed610c4332b6ba59a75414ff84"
format_stamp: "Formatted at 2026-09-28 06:27:40 on dist-test-slave-q03f"
server_key: "e832315e2a6cdb169b8f89fcbec21991"
server_key_iv: "7138838ddbf161667b0e8a817cb72921"
server_key_version: "encryptionkey@0"
I20260928 06:27:40.468436 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "6597e5ed610c4332b6ba59a75414ff84"
format_stamp: "Formatted at 2026-09-28 06:27:40 on dist-test-slave-q03f"
server_key: "e832315e2a6cdb169b8f89fcbec21991"
server_key_iv: "7138838ddbf161667b0e8a817cb72921"
server_key_version: "encryptionkey@0"
I20260928 06:27:40.469913 21032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:27:40.471071 21047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:40.471414 21032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:40.471534 21032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "6597e5ed610c4332b6ba59a75414ff84"
format_stamp: "Formatted at 2026-09-28 06:27:40 on dist-test-slave-q03f"
server_key: "e832315e2a6cdb169b8f89fcbec21991"
server_key_iv: "7138838ddbf161667b0e8a817cb72921"
server_key_version: "encryptionkey@0"
I20260928 06:27:40.471618 21032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:40.480607 21032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:40.483084 21032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:40.483223 21032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:40.487668 21032 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40361
I20260928 06:27:40.487722 21099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40361 every 8 connection(s)
I20260928 06:27:40.488180 21032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:40.489283 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:40.492000 21100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84: Bootstrap starting.
I20260928 06:27:40.492329 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 21032
I20260928 06:27:40.492491 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:40.492703 32601 external_mini_cluster.cc:1428] Setting key c2181b740046f13cb1a5a3d694e833bb
I20260928 06:27:40.492830 21100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:40.493196 21100 log.cc:824] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:40.494217 21100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84: No bootstrap required, opened a new log
I20260928 06:27:40.496647 21100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597e5ed610c4332b6ba59a75414ff84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40361 } }
I20260928 06:27:40.496855 21100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:40.496897 21100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6597e5ed610c4332b6ba59a75414ff84, State: Initialized, Role: FOLLOWER
I20260928 06:27:40.497043 21100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597e5ed610c4332b6ba59a75414ff84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40361 } }
I20260928 06:27:40.497123 21100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:40.497145 21100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:40.497218 21100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:40.497908 21100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597e5ed610c4332b6ba59a75414ff84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40361 } }
I20260928 06:27:40.498059 21100 leader_election.cc:302] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6597e5ed610c4332b6ba59a75414ff84; no voters:
I20260928 06:27:40.498231 21100 leader_election.cc:288] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:40.498322 21105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:40.498524 21105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [term 1 LEADER]: Becoming Leader. State: Replica: 6597e5ed610c4332b6ba59a75414ff84, State: Running, Role: LEADER
I20260928 06:27:40.498575 21100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:40.499436 21105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597e5ed610c4332b6ba59a75414ff84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40361 } }
I20260928 06:27:40.500037 21107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6597e5ed610c4332b6ba59a75414ff84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597e5ed610c4332b6ba59a75414ff84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40361 } } }
I20260928 06:27:40.500134 21107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:40.500542 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6597e5ed610c4332b6ba59a75414ff84. Latest consensus state: current_term: 1 leader_uuid: "6597e5ed610c4332b6ba59a75414ff84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597e5ed610c4332b6ba59a75414ff84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40361 } } }
I20260928 06:27:40.500619 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84 [sys.catalog]: This master's current role is: LEADER
Sep 28 06:27:40 dist-test-slave-q03f krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260928 06:27:40.501291 21119 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:27:40.501380 21119 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:27:40.501451 21120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:40.501647 21120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:40.503501 21120 catalog_manager.cc:1347] Generated new cluster ID: 9d470c5f374b442a840a7e1018c76549
I20260928 06:27:40.503556 21120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:40.506076 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:40.493327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:34587 (local address 127.0.0.1:40361)
0928 06:27:40.493543 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:40.493553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:40.493590 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:27:40.494298 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:40.494308 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:40.494390 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:40.494566 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:40.495251 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:40.496735 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:40.497876 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:40.498566 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:40.499056 (+ 490us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:40.501912 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:40.501916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:40.501916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:40.501930 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:40.504214 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:40.504702 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:40.504706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:40.504706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:40.504830 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:40.505083 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:40.505084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:40.505085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:40.505408 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:40.505581 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:40.505682 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:27:40.505806 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":109,"threads_started":1}
I20260928 06:27:40.507143 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:40.510916 21120 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:40.511565 21120 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:40.516183 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:40.508602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:36713 (local address 127.0.0.1:40361)
0928 06:27:40.508636 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:40.508638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:40.508756 (+ 118us) server_negotiation.cc:407] Connection header received
0928 06:27:40.508828 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:40.508829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:40.508839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:40.508897 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:40.509532 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:40.510297 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:40.511356 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:40.511675 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:40.511749 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:40.513212 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:40.513213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:40.513214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:40.513220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:40.514938 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:40.515362 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:40.515363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:40.515364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:40.515418 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:40.515645 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:40.515645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:40.515646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:40.515763 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:40.515872 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:40.516028 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:27:40.516092 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:27:40.516897 21120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6597e5ed610c4332b6ba59a75414ff84: Generated new TSK 0
I20260928 06:27:40.517062 21120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:40.523713 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 21032
W20260928 06:27:40.528311 21017 connection.cc:535] client connection to 127.0.0.1:40361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40361): Connection reset by peer (error 104)
2026-09-28T06:27:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:40 dist-test-slave-q03f krb5kdc[21131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:40 dist-test-slave-q03f krb5kdc[21131](info): set up 2 sockets
Sep 28 06:27:40 dist-test-slave-q03f krb5kdc[21131](info): commencing operation
krb5kdc: starting...
W20260928 06:27:41.566520 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:41 dist-test-slave-q03f krb5kdc[21131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:41.702745 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:41.788182 21144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:41.788470 21144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:41.788513 21144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:41.789736 21144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:41.789908 21144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:41.789983 21144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:41.790014 21144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:41.790040 21144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:41.791615 21144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:41.791925 21144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:41.792176 21144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:41.799504 21149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:41.799533 21150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:41.799517 21152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:41.799552 21144 server_base.cc:1034] running on GCE node
I20260928 06:27:41.800165 21144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:41.800494 21144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:41.801673 21144 hybrid_clock.cc:648] HybridClock initialized: now 1790576861801629 us; error 66 us; skew 500 ppm
Sep 28 06:27:41 dist-test-slave-q03f krb5kdc[21131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:41.804466 21144 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:41.805058 21144 webserver.cc:460] Webserver started at http://127.0.0.1:33067/ using document root <none> and password file <none>
I20260928 06:27:41.805301 21144 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:41.805373 21144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:41.805536 21144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:41.806856 21144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "51101257b2ea4b0cbaa496f303a322bb"
format_stamp: "Formatted at 2026-09-28 06:27:41 on dist-test-slave-q03f"
server_key: "cab75f9b212f4e1564e177c2e0e64ee9"
server_key_iv: "3635d52bbd56a59c160e694640a11366"
server_key_version: "encryptionkey@0"
I20260928 06:27:41.807245 21144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "51101257b2ea4b0cbaa496f303a322bb"
format_stamp: "Formatted at 2026-09-28 06:27:41 on dist-test-slave-q03f"
server_key: "cab75f9b212f4e1564e177c2e0e64ee9"
server_key_iv: "3635d52bbd56a59c160e694640a11366"
server_key_version: "encryptionkey@0"
I20260928 06:27:41.808730 21144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:27:41.809996 21159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:41.810292 21144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:41.810428 21144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "51101257b2ea4b0cbaa496f303a322bb"
format_stamp: "Formatted at 2026-09-28 06:27:41 on dist-test-slave-q03f"
server_key: "cab75f9b212f4e1564e177c2e0e64ee9"
server_key_iv: "3635d52bbd56a59c160e694640a11366"
server_key_version: "encryptionkey@0"
I20260928 06:27:41.810521 21144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:41.832104 21144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:41.834820 21144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:41.835059 21144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:41.839504 21144 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34433
I20260928 06:27:41.839533 21211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34433 every 8 connection(s)
I20260928 06:27:41.839948 21144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:41.840898 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:41.843439 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb: Bootstrap starting.
I20260928 06:27:41.844084 21212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:41.844393 21212 log.cc:824] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:41.845465 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb: No bootstrap required, opened a new log
I20260928 06:27:41.847354 21212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51101257b2ea4b0cbaa496f303a322bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34433 } }
I20260928 06:27:41.847505 21212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:41.847529 21212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51101257b2ea4b0cbaa496f303a322bb, State: Initialized, Role: FOLLOWER
I20260928 06:27:41.847710 21212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51101257b2ea4b0cbaa496f303a322bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34433 } }
I20260928 06:27:41.847780 21212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:41.847824 21212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:41.847890 21212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:41.848505 21212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51101257b2ea4b0cbaa496f303a322bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34433 } }
I20260928 06:27:41.848621 21212 leader_election.cc:302] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51101257b2ea4b0cbaa496f303a322bb; no voters:
I20260928 06:27:41.848837 21212 leader_election.cc:288] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:41.848968 21215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:41.849131 21212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:41.849116 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 21144
I20260928 06:27:41.849176 21215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [term 1 LEADER]: Becoming Leader. State: Replica: 51101257b2ea4b0cbaa496f303a322bb, State: Running, Role: LEADER
I20260928 06:27:41.849251 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:41.849452 32601 external_mini_cluster.cc:1428] Setting key e09d75b10b05643f4ecb5de8cacc64c3
I20260928 06:27:41.849356 21215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51101257b2ea4b0cbaa496f303a322bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34433 } }
I20260928 06:27:41.850069 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51101257b2ea4b0cbaa496f303a322bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51101257b2ea4b0cbaa496f303a322bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34433 } } }
I20260928 06:27:41.850226 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:41.850534 21216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51101257b2ea4b0cbaa496f303a322bb. Latest consensus state: current_term: 1 leader_uuid: "51101257b2ea4b0cbaa496f303a322bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51101257b2ea4b0cbaa496f303a322bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34433 } } }
I20260928 06:27:41.850661 21216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:41.850791 21226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:41.851491 21226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:41.854410 21226 catalog_manager.cc:1347] Generated new cluster ID: e7e1dfee1d9b4871a80af3bcee0415b1
I20260928 06:27:41.854475 21226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:41.856334 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:41.850251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:51203 (local address 127.0.0.1:34433)
0928 06:27:41.850749 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:41.850761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:41.850804 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:27:41.850941 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:41.850949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:41.851020 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:41.851237 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:41.851539 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:41.853034 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:41.853940 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:41.854638 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:41.854763 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:41.855588 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:41.855592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:41.855592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:41.855833 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:41.855873 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:41.855889 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:41.855982 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:41.856074 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 06:27:41.856105 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":88,"threads_started":1}
I20260928 06:27:41.859180 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:41.864804 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:41.860951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:58749 (local address 127.0.0.1:34433)
0928 06:27:41.861007 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:41.861009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:41.861018 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:27:41.861040 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:41.861041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:41.861052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:41.861122 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:41.861307 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:41.862254 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:41.863340 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:41.863662 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:41.863726 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:41.864503 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:41.864505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:41.864505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:41.864509 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:41.864524 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:41.864528 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:41.864584 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:41.864699 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:27:41.864712 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:27:41.869367 21226 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:41.870038 21226 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:41.872447 21226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51101257b2ea4b0cbaa496f303a322bb: Generated new TSK 0
I20260928 06:27:41.872649 21226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:41.876861 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 21144
W20260928 06:27:41.881718 21129 connection.cc:535] client connection to 127.0.0.1:34433 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34433): Connection reset by peer (error 104)
2026-09-28T06:27:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:41 dist-test-slave-q03f krb5kdc[21243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:41 dist-test-slave-q03f krb5kdc[21243](info): set up 2 sockets
Sep 28 06:27:41 dist-test-slave-q03f krb5kdc[21243](info): commencing operation
krb5kdc: starting...
W20260928 06:27:42.990689 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:43 dist-test-slave-q03f krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:43.126042 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:43.211575 21256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:43.211786 21256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:43.211812 21256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:43.212996 21256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:43.213044 21256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:43.213058 21256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:43.213071 21256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:43.213083 21256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:43.214654 21256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:43.214933 21256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:43.215174 21256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:27:43.220657 21256 server_base.cc:1034] running on GCE node
W20260928 06:27:43.220616 21262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:43.220594 21264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:43.220607 21261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:43.221266 21256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:43.221616 21256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:43.222858 21256 hybrid_clock.cc:648] HybridClock initialized: now 1790576863222849 us; error 46 us; skew 500 ppm
Sep 28 06:27:43 dist-test-slave-q03f krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:43.233107 21256 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:43.233702 21256 webserver.cc:460] Webserver started at http://127.0.0.1:36551/ using document root <none> and password file <none>
I20260928 06:27:43.234038 21256 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:43.234124 21256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:43.234323 21256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:43.235586 21256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "d8b1194e13a34f53b679543f8a20b3a7"
format_stamp: "Formatted at 2026-09-28 06:27:43 on dist-test-slave-q03f"
server_key: "32d027c20a7af6356f6e9611d03e47f3"
server_key_iv: "efcace17767503a86f60dbed967dbe03"
server_key_version: "encryptionkey@0"
I20260928 06:27:43.236027 21256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "d8b1194e13a34f53b679543f8a20b3a7"
format_stamp: "Formatted at 2026-09-28 06:27:43 on dist-test-slave-q03f"
server_key: "32d027c20a7af6356f6e9611d03e47f3"
server_key_iv: "efcace17767503a86f60dbed967dbe03"
server_key_version: "encryptionkey@0"
I20260928 06:27:43.237658 21256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:27:43.238812 21271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:43.239090 21256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:43.239219 21256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "d8b1194e13a34f53b679543f8a20b3a7"
format_stamp: "Formatted at 2026-09-28 06:27:43 on dist-test-slave-q03f"
server_key: "32d027c20a7af6356f6e9611d03e47f3"
server_key_iv: "efcace17767503a86f60dbed967dbe03"
server_key_version: "encryptionkey@0"
I20260928 06:27:43.239324 21256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:43.243335 21256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:43.246076 21256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:43.246265 21256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:43.251036 21256 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34801
I20260928 06:27:43.251147 21323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34801 every 8 connection(s)
I20260928 06:27:43.251667 21256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:43.252852 21324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:43.255300 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7: Bootstrap starting.
I20260928 06:27:43.255867 21324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:43.256111 21324 log.cc:824] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:43.256824 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7: No bootstrap required, opened a new log
I20260928 06:27:43.258682 21324 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b1194e13a34f53b679543f8a20b3a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260928 06:27:43.258834 21324 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:43.258906 21324 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b1194e13a34f53b679543f8a20b3a7, State: Initialized, Role: FOLLOWER
I20260928 06:27:43.259078 21324 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b1194e13a34f53b679543f8a20b3a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260928 06:27:43.259166 21324 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:43.259192 21324 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:43.259222 21324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:43.259756 21324 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b1194e13a34f53b679543f8a20b3a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260928 06:27:43.259850 21324 leader_election.cc:302] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8b1194e13a34f53b679543f8a20b3a7; no voters:
I20260928 06:27:43.260035 21324 leader_election.cc:288] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:43.260063 21327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:43.260182 21327 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [term 1 LEADER]: Becoming Leader. State: Replica: d8b1194e13a34f53b679543f8a20b3a7, State: Running, Role: LEADER
I20260928 06:27:43.260322 21324 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:43.260329 21327 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b1194e13a34f53b679543f8a20b3a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } }
I20260928 06:27:43.260819 21329 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8b1194e13a34f53b679543f8a20b3a7. Latest consensus state: current_term: 1 leader_uuid: "d8b1194e13a34f53b679543f8a20b3a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b1194e13a34f53b679543f8a20b3a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } } }
I20260928 06:27:43.260926 21329 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:43.260970 21328 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8b1194e13a34f53b679543f8a20b3a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b1194e13a34f53b679543f8a20b3a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34801 } } }
I20260928 06:27:43.261065 21328 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:43.261261 21335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:43.261641 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 21256
I20260928 06:27:43.261777 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:43.261924 21335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:43.262034 32601 external_mini_cluster.cc:1428] Setting key 18fa0de82050dc1f4544bc3bfa146dd9
I20260928 06:27:43.264735 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:43.262807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:37433 (local address 127.0.0.1:34801)
0928 06:27:43.263018 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:43.263030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:43.263068 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:27:43.263200 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:43.263207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:43.263270 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:43.263437 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:43.264230 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:43.264233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:43.264234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:43.264238 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:43.264276 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:43.264289 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:43.264352 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:43.264454 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:27:43.264519 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260928 06:27:43.265887 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:43.268654 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:43.267482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:59791 (local address 127.0.0.1:34801)
0928 06:27:43.267531 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:43.267534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:43.267589 (+ 55us) server_negotiation.cc:407] Connection header received
0928 06:27:43.267671 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:43.267672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:43.267682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:43.267736 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:43.268436 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:43.268437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:43.268437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:43.268441 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:43.268459 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:43.268463 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:43.268534 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:43.268577 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 06:27:43.268590 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:27:43.270464 21335 catalog_manager.cc:1347] Generated new cluster ID: 68bd2090160442a78eaecfc4a75275d9
I20260928 06:27:43.270525 21335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:43.281354 21335 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:43.281612 21335 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:43.283560 21335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8b1194e13a34f53b679543f8a20b3a7: Generated new TSK 0
I20260928 06:27:43.283790 21335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:43.287686 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 21256
2026-09-28T06:27:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:43 dist-test-slave-q03f krb5kdc[21355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:43 dist-test-slave-q03f krb5kdc[21355](info): set up 2 sockets
Sep 28 06:27:43 dist-test-slave-q03f krb5kdc[21355](info): commencing operation
krb5kdc: starting...
W20260928 06:27:44.386363 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:44 dist-test-slave-q03f krb5kdc[21355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:44.513918 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:44.600618 21368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:44.600903 21368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:44.600948 21368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:44.602232 21368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:44.602312 21368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:44.602336 21368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:44.602357 21368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:44.602388 21368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:44.604003 21368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:44.604315 21368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:44.604562 21368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:44.609696 21374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:44.609696 21376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:44.609851 21373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:44.610291 21368 server_base.cc:1034] running on GCE node
I20260928 06:27:44.610582 21368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:44.611014 21368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:44.612210 21368 hybrid_clock.cc:648] HybridClock initialized: now 1790576864612183 us; error 42 us; skew 500 ppm
Sep 28 06:27:44 dist-test-slave-q03f krb5kdc[21355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:44.615022 21368 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:44.615517 21368 webserver.cc:460] Webserver started at http://127.0.0.1:32849/ using document root <none> and password file <none>
I20260928 06:27:44.615720 21368 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:44.615762 21368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:44.615891 21368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:44.616973 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "3f1160a51b70444eabd50255751ee132"
format_stamp: "Formatted at 2026-09-28 06:27:44 on dist-test-slave-q03f"
server_key: "e92361e4672144a7485294f23315977f"
server_key_iv: "166528c181accefad92e4c71750c9fb6"
server_key_version: "encryptionkey@0"
I20260928 06:27:44.617282 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "3f1160a51b70444eabd50255751ee132"
format_stamp: "Formatted at 2026-09-28 06:27:44 on dist-test-slave-q03f"
server_key: "e92361e4672144a7485294f23315977f"
server_key_iv: "166528c181accefad92e4c71750c9fb6"
server_key_version: "encryptionkey@0"
I20260928 06:27:44.618741 21368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:27:44.619959 21383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:44.620324 21368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:44.620437 21368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "3f1160a51b70444eabd50255751ee132"
format_stamp: "Formatted at 2026-09-28 06:27:44 on dist-test-slave-q03f"
server_key: "e92361e4672144a7485294f23315977f"
server_key_iv: "166528c181accefad92e4c71750c9fb6"
server_key_version: "encryptionkey@0"
I20260928 06:27:44.620554 21368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:44.646553 21368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:44.649582 21368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:44.649791 21368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:44.654644 21368 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40857
I20260928 06:27:44.654670 21435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40857 every 8 connection(s)
I20260928 06:27:44.655166 21368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:44.656158 21436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:44.658604 21436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132: Bootstrap starting.
I20260928 06:27:44.659255 21436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:44.659550 21436 log.cc:824] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:44.659910 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 21368
I20260928 06:27:44.660034 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:44.660265 32601 external_mini_cluster.cc:1428] Setting key c3094bce4d0b6e8d6278bed8193fbd55
I20260928 06:27:44.660502 21436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132: No bootstrap required, opened a new log
I20260928 06:27:44.662315 21436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1160a51b70444eabd50255751ee132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260928 06:27:44.662487 21436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:44.662510 21436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1160a51b70444eabd50255751ee132, State: Initialized, Role: FOLLOWER
I20260928 06:27:44.662746 21436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1160a51b70444eabd50255751ee132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260928 06:27:44.662846 21436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:44.662895 21436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:44.662945 21436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:44.663830 21436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1160a51b70444eabd50255751ee132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260928 06:27:44.663951 21436 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1160a51b70444eabd50255751ee132; no voters:
I20260928 06:27:44.664179 21436 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:44.664230 21441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:44.664390 21441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [term 1 LEADER]: Becoming Leader. State: Replica: 3f1160a51b70444eabd50255751ee132, State: Running, Role: LEADER
I20260928 06:27:44.664467 21436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:44.664538 21441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1160a51b70444eabd50255751ee132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } }
I20260928 06:27:44.664873 21441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1160a51b70444eabd50255751ee132. Latest consensus state: current_term: 1 leader_uuid: "3f1160a51b70444eabd50255751ee132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1160a51b70444eabd50255751ee132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } } }
I20260928 06:27:44.664973 21441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:44.664887 21442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f1160a51b70444eabd50255751ee132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1160a51b70444eabd50255751ee132" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40857 } } }
I20260928 06:27:44.665127 21442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:44.665472 21449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:44.666087 21449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:44.667894 21449 catalog_manager.cc:1347] Generated new cluster ID: 5837a9779726423dac8a494b872dcf33
I20260928 06:27:44.667968 21449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:27:44 dist-test-slave-q03f krb5kdc[21355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:27:44.673727 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:44.660922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:59117 (local address 127.0.0.1:40857)
0928 06:27:44.661084 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:44.661094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:44.661200 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:27:44.662307 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:44.662316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:44.662376 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:44.662540 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:44.663391 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:44.664487 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:44.665675 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:44.666304 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:44.666810 (+ 506us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:44.669388 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:44.669392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:44.669393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:44.669407 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:44.671819 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:44.672285 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:44.672289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:44.672290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:44.672442 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:44.672694 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:44.672695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:44.672695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:44.673064 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:44.673283 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:44.673376 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:27:44.673493 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260928 06:27:44.675154 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:44.679373 21449 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:44.679957 21449 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:44.684927 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:44.676910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:34219 (local address 127.0.0.1:40857)
0928 06:27:44.676944 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:44.676946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:44.677058 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:27:44.677139 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:44.677140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:44.677151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:44.677201 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:44.677911 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:44.678675 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:44.679813 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:44.680128 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:44.680196 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:44.681801 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:44.681803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:44.681803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:44.681811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:44.683502 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:44.683972 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:44.683973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:44.683973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:44.684030 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:44.684290 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:44.684290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:44.684291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:44.684412 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:44.684522 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:44.684693 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:27:44.684756 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:27:44.688868 21449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f1160a51b70444eabd50255751ee132: Generated new TSK 0
I20260928 06:27:44.689082 21449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:44.693387 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 21368
W20260928 06:27:44.698009 21353 connection.cc:535] client connection to 127.0.0.1:40857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40857): Connection reset by peer (error 104)
2026-09-28T06:27:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:44 dist-test-slave-q03f krb5kdc[21467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:44 dist-test-slave-q03f krb5kdc[21467](info): set up 2 sockets
Sep 28 06:27:44 dist-test-slave-q03f krb5kdc[21467](info): commencing operation
krb5kdc: starting...
W20260928 06:27:45.738929 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:45 dist-test-slave-q03f krb5kdc[21467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:45.886250 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:45.972885 21480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:45.973117 21480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:45.973142 21480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:45.974481 21480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:45.974536 21480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:45.974551 21480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:45.974565 21480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:45.974576 21480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:45.976240 21480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:45.976543 21480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:45.976802 21480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:45.982270 21486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:45.982277 21488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:45.982277 21485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:45.982645 21480 server_base.cc:1034] running on GCE node
I20260928 06:27:45.982931 21480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:45.983258 21480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:45.984491 21480 hybrid_clock.cc:648] HybridClock initialized: now 1790576865984483 us; error 38 us; skew 500 ppm
Sep 28 06:27:45 dist-test-slave-q03f krb5kdc[21467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:45.987286 21480 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:45.987720 21480 webserver.cc:460] Webserver started at http://127.0.0.1:36945/ using document root <none> and password file <none>
I20260928 06:27:45.987967 21480 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:45.988014 21480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:45.988147 21480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:45.989300 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "c6125b64530f417eb29eade5f0a710e2"
format_stamp: "Formatted at 2026-09-28 06:27:45 on dist-test-slave-q03f"
server_key: "fab488e40132830f5d3395df08aed447"
server_key_iv: "8ab1d5bf763093a7258ceabce7fbde64"
server_key_version: "encryptionkey@0"
I20260928 06:27:45.989672 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "c6125b64530f417eb29eade5f0a710e2"
format_stamp: "Formatted at 2026-09-28 06:27:45 on dist-test-slave-q03f"
server_key: "fab488e40132830f5d3395df08aed447"
server_key_iv: "8ab1d5bf763093a7258ceabce7fbde64"
server_key_version: "encryptionkey@0"
I20260928 06:27:45.991240 21480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:27:45.992132 21495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:45.992347 21480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:45.992477 21480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "c6125b64530f417eb29eade5f0a710e2"
format_stamp: "Formatted at 2026-09-28 06:27:45 on dist-test-slave-q03f"
server_key: "fab488e40132830f5d3395df08aed447"
server_key_iv: "8ab1d5bf763093a7258ceabce7fbde64"
server_key_version: "encryptionkey@0"
I20260928 06:27:45.992555 21480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:46.009224 21480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:46.012105 21480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:46.012332 21480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:46.016753 21480 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37677
I20260928 06:27:46.016884 21547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37677 every 8 connection(s)
I20260928 06:27:46.017261 21480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:46.018373 21548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:46.021101 21548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2: Bootstrap starting.
I20260928 06:27:46.021775 21548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:46.022130 21548 log.cc:824] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:46.022960 21548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2: No bootstrap required, opened a new log
I20260928 06:27:46.023099 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 21480
I20260928 06:27:46.023242 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:46.023438 32601 external_mini_cluster.cc:1428] Setting key d09ea2ce2b18a9257719bff52284fe6d
I20260928 06:27:46.025334 21548 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6125b64530f417eb29eade5f0a710e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } }
I20260928 06:27:46.025528 21548 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:46.025574 21548 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6125b64530f417eb29eade5f0a710e2, State: Initialized, Role: FOLLOWER
I20260928 06:27:46.025802 21548 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6125b64530f417eb29eade5f0a710e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } }
I20260928 06:27:46.025911 21548 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:46.025995 21548 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:46.026055 21548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:46.026750 21548 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6125b64530f417eb29eade5f0a710e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } }
I20260928 06:27:46.026849 21548 leader_election.cc:302] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6125b64530f417eb29eade5f0a710e2; no voters:
I20260928 06:27:46.027091 21548 leader_election.cc:288] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:46.027132 21553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:46.027263 21553 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [term 1 LEADER]: Becoming Leader. State: Replica: c6125b64530f417eb29eade5f0a710e2, State: Running, Role: LEADER
I20260928 06:27:46.027418 21553 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6125b64530f417eb29eade5f0a710e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } }
I20260928 06:27:46.027420 21548 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:46.029513 21555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6125b64530f417eb29eade5f0a710e2. Latest consensus state: current_term: 1 leader_uuid: "c6125b64530f417eb29eade5f0a710e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6125b64530f417eb29eade5f0a710e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } } }
I20260928 06:27:46.029538 21554 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6125b64530f417eb29eade5f0a710e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6125b64530f417eb29eade5f0a710e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } } }
W20260928 06:27:46.029752 21568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 06:27:46.030053 21555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [sys.catalog]: This master's current role is: LEADER
W20260928 06:27:46.030109 21568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:27:46.030305 21570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:46.030510 21570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:46.031312 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:46.024062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:53067 (local address 127.0.0.1:37677)
0928 06:27:46.024269 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:46.024278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:46.024304 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:27:46.025320 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:46.025330 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:46.025393 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:46.025554 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:46.025923 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:46.027561 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:46.028749 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:46.029220 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:46.029337 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:46.030416 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:46.030420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:46.030420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:46.030690 (+ 270us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:46.030731 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:46.030746 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:46.030830 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:46.031033 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 06:27:46.031062 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260928 06:27:46.030063 21554 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:46.032436 21570 catalog_manager.cc:1347] Generated new cluster ID: d496cbd0a1954df28f2d884cbeb76d9c
I20260928 06:27:46.032464 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:46.032486 21570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:46.042243 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:46.034024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:43813 (local address 127.0.0.1:37677)
0928 06:27:46.034058 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:46.034060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:46.038238 (+ 4178us) server_negotiation.cc:407] Connection header received
0928 06:27:46.038319 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:46.038320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:46.038334 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:46.038405 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:46.038619 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:46.039370 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:46.040618 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:46.040957 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:46.041023 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:46.041908 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:46.041909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:46.041910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:46.041916 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:46.041932 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:46.041936 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:46.042012 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:46.042117 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:27:46.042131 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:27:46.052248 21570 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:46.053829 21570 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:46.059525 21570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6125b64530f417eb29eade5f0a710e2: Generated new TSK 0
I20260928 06:27:46.059770 21570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:46.064000 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 21480
W20260928 06:27:46.069164 21465 connection.cc:535] client connection to 127.0.0.1:37677 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37677): Connection reset by peer (error 104)
2026-09-28T06:27:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:46 dist-test-slave-q03f krb5kdc[21579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:46 dist-test-slave-q03f krb5kdc[21579](info): set up 2 sockets
Sep 28 06:27:46 dist-test-slave-q03f krb5kdc[21579](info): commencing operation
krb5kdc: starting...
W20260928 06:27:47.114239 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:47 dist-test-slave-q03f krb5kdc[21579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:27:47.245121 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:47.328295 21592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:47.328572 21592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:47.328614 21592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:47.329818 21592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:47.329888 21592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:47.329912 21592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:47.329936 21592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:47.330001 21592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:47.331532 21592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:47.331833 21592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:47.332073 21592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:47.337162 21597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:47.337239 21598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:47.337146 21600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:47.337543 21592 server_base.cc:1034] running on GCE node
I20260928 06:27:47.337881 21592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:47.338214 21592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:47.339387 21592 hybrid_clock.cc:648] HybridClock initialized: now 1790576867339371 us; error 27 us; skew 500 ppm
Sep 28 06:27:47 dist-test-slave-q03f krb5kdc[21579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:47.342182 21592 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:27:47.342634 21592 webserver.cc:460] Webserver started at http://127.0.0.1:40941/ using document root <none> and password file <none>
I20260928 06:27:47.342905 21592 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:47.342957 21592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:47.343106 21592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:47.344244 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "cabd12dec4514b3e86855406aebeb70e"
format_stamp: "Formatted at 2026-09-28 06:27:47 on dist-test-slave-q03f"
server_key: "086c4d5f4f9b327f8d19aae2b0448be8"
server_key_iv: "1271a5eb75053628c8059f04f070fbc1"
server_key_version: "encryptionkey@0"
I20260928 06:27:47.344602 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "cabd12dec4514b3e86855406aebeb70e"
format_stamp: "Formatted at 2026-09-28 06:27:47 on dist-test-slave-q03f"
server_key: "086c4d5f4f9b327f8d19aae2b0448be8"
server_key_iv: "1271a5eb75053628c8059f04f070fbc1"
server_key_version: "encryptionkey@0"
I20260928 06:27:47.346076 21592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:27:47.346863 21607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:47.347036 21592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:27:47.347136 21592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "cabd12dec4514b3e86855406aebeb70e"
format_stamp: "Formatted at 2026-09-28 06:27:47 on dist-test-slave-q03f"
server_key: "086c4d5f4f9b327f8d19aae2b0448be8"
server_key_iv: "1271a5eb75053628c8059f04f070fbc1"
server_key_version: "encryptionkey@0"
I20260928 06:27:47.347204 21592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:47.350615 21592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:47.352752 21592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:47.352866 21592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:47.357066 21592 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45963
I20260928 06:27:47.357103 21659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45963 every 8 connection(s)
I20260928 06:27:47.357467 21592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:47.358553 21660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:47.360320 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 21592
I20260928 06:27:47.360419 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:47.360697 32601 external_mini_cluster.cc:1428] Setting key 2246677565b11855a73380c89a6ea1c2
I20260928 06:27:47.361699 21660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e: Bootstrap starting.
I20260928 06:27:47.362552 21660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:47.362936 21660 log.cc:824] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:47.363842 21660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e: No bootstrap required, opened a new log
I20260928 06:27:47.364027 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:47.361373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:60255 (local address 127.0.0.1:45963)
0928 06:27:47.361674 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:47.361685 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:47.361721 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:27:47.362409 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:47.362420 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:47.362467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:47.362656 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:47.363454 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:47.363459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:47.363459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:47.363464 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:47.363516 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:47.363537 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:47.363623 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:47.363701 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:27:47.363765 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":74,"threads_started":1}
I20260928 06:27:47.365103 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:47.366359 21660 raft_consensus.cc:348] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd12dec4514b3e86855406aebeb70e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260928 06:27:47.366593 21660 raft_consensus.cc:374] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:47.366641 21660 raft_consensus.cc:729] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabd12dec4514b3e86855406aebeb70e, State: Initialized, Role: FOLLOWER
I20260928 06:27:47.366789 21660 consensus_queue.cc:260] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd12dec4514b3e86855406aebeb70e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260928 06:27:47.366904 21660 raft_consensus.cc:388] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:47.366968 21660 raft_consensus.cc:482] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:47.367024 21660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:47.367659 21660 raft_consensus.cc:504] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd12dec4514b3e86855406aebeb70e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260928 06:27:47.367785 21660 leader_election.cc:302] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cabd12dec4514b3e86855406aebeb70e; no voters:
I20260928 06:27:47.367987 21660 leader_election.cc:288] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:47.368012 21670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:47.368110 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:47.366900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.147:50267 (local address 127.0.0.1:45963)
0928 06:27:47.366956 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:47.366958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:47.367081 (+ 123us) server_negotiation.cc:407] Connection header received
0928 06:27:47.367185 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:47.367185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:47.367190 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:47.367245 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:47.367893 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:47.367894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:47.367894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:47.367898 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:47.367912 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:47.367916 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:47.367966 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:47.368019 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:27:47.368034 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:27:47.368199 21670 raft_consensus.cc:686] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [term 1 LEADER]: Becoming Leader. State: Replica: cabd12dec4514b3e86855406aebeb70e, State: Running, Role: LEADER
I20260928 06:27:47.368278 21670 consensus_queue.cc:237] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd12dec4514b3e86855406aebeb70e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } }
I20260928 06:27:47.368312 21660 sys_catalog.cc:564] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:47.368685 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cabd12dec4514b3e86855406aebeb70e. Latest consensus state: current_term: 1 leader_uuid: "cabd12dec4514b3e86855406aebeb70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd12dec4514b3e86855406aebeb70e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } } }
I20260928 06:27:47.368686 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cabd12dec4514b3e86855406aebeb70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabd12dec4514b3e86855406aebeb70e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45963 } } }
I20260928 06:27:47.368814 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:47.368871 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:47.369606 21676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:47.370270 21676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:47.371825 21676 catalog_manager.cc:1347] Generated new cluster ID: 76985f1ace914bc9bfe7fadfe7aac99c
I20260928 06:27:47.371893 21676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:47.404227 21676 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:47.404471 21676 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:47.409238 21676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cabd12dec4514b3e86855406aebeb70e: Generated new TSK 0
I20260928 06:27:47.409471 21676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:47.412417 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 21592
2026-09-28T06:27:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:47 dist-test-slave-q03f krb5kdc[21691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:47 dist-test-slave-q03f krb5kdc[21691](info): set up 2 sockets
Sep 28 06:27:47 dist-test-slave-q03f krb5kdc[21691](info): commencing operation
krb5kdc: starting...
W20260928 06:27:48.510489 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:48 dist-test-slave-q03f krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:48.646596 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38059
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:38059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:48.733150 21704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:48.733378 21704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:48.733404 21704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:48.734632 21704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:48.734678 21704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:48.734699 21704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:48.734711 21704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:48.734723 21704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:48.736213 21704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:38059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:48.736438 21704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:48.736635 21704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:48.742425 21710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:48.742425 21712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:48.742502 21709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:48.742660 21704 server_base.cc:1034] running on GCE node
I20260928 06:27:48.743036 21704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:48.743321 21704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:48.744469 21704 hybrid_clock.cc:648] HybridClock initialized: now 1790576868744423 us; error 67 us; skew 500 ppm
Sep 28 06:27:48 dist-test-slave-q03f krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:48.754374 21704 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:48.754935 21704 webserver.cc:460] Webserver started at http://127.31.214.126:40001/ using document root <none> and password file <none>
I20260928 06:27:48.755191 21704 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:48.755249 21704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:48.755426 21704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:48.756619 21704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6"
format_stamp: "Formatted at 2026-09-28 06:27:48 on dist-test-slave-q03f"
server_key: "48f553e66cb41adb3fb7e5c35eeee92c"
server_key_iv: "3445c37a7b30a0363c1edc755ef7d4dc"
server_key_version: "encryptionkey@0"
I20260928 06:27:48.757015 21704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6"
format_stamp: "Formatted at 2026-09-28 06:27:48 on dist-test-slave-q03f"
server_key: "48f553e66cb41adb3fb7e5c35eeee92c"
server_key_iv: "3445c37a7b30a0363c1edc755ef7d4dc"
server_key_version: "encryptionkey@0"
I20260928 06:27:48.758651 21704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:27:48.759512 21719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:48.759716 21704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:48.759807 21704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6"
format_stamp: "Formatted at 2026-09-28 06:27:48 on dist-test-slave-q03f"
server_key: "48f553e66cb41adb3fb7e5c35eeee92c"
server_key_iv: "3445c37a7b30a0363c1edc755ef7d4dc"
server_key_version: "encryptionkey@0"
I20260928 06:27:48.759909 21704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:48.780431 21704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:48.782778 21704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:48.782936 21704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:48.787294 21704 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:38059
I20260928 06:27:48.787336 21771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:38059 every 8 connection(s)
I20260928 06:27:48.787742 21704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:48.788738 21772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:48.791512 21772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6: Bootstrap starting.
I20260928 06:27:48.792215 21772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:48.792541 21772 log.cc:824] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:48.792640 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 21704
I20260928 06:27:48.792850 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:48.793102 32601 external_mini_cluster.cc:1428] Setting key 62df79cc469e30f1159dcfe974c4c306
I20260928 06:27:48.793510 21772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6: No bootstrap required, opened a new log
I20260928 06:27:48.795500 21772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38059 } }
I20260928 06:27:48.795689 21772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:48.795763 21772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d1c1fe7bdc34f92ab54ba4bf9e22da6, State: Initialized, Role: FOLLOWER
I20260928 06:27:48.795928 21772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38059 } }
I20260928 06:27:48.796041 21772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:48.796097 21772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:48.796151 21772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:48.797143 21772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38059 } }
I20260928 06:27:48.797276 21772 leader_election.cc:302] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d1c1fe7bdc34f92ab54ba4bf9e22da6; no voters:
I20260928 06:27:48.797474 21772 leader_election.cc:288] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:48.797523 21777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:48.797679 21777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [term 1 LEADER]: Becoming Leader. State: Replica: 2d1c1fe7bdc34f92ab54ba4bf9e22da6, State: Running, Role: LEADER
I20260928 06:27:48.797776 21772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:48.797827 21777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38059 } }
I20260928 06:27:48.798249 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d1c1fe7bdc34f92ab54ba4bf9e22da6. Latest consensus state: current_term: 1 leader_uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38059 } } }
I20260928 06:27:48.798329 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:48.798619 21778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1c1fe7bdc34f92ab54ba4bf9e22da6" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38059 } } }
I20260928 06:27:48.798755 21778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:48.799120 21782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:48.799782 21782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:48.801842 21782 catalog_manager.cc:1347] Generated new cluster ID: 1b03bbcf60174b04a71bb6735b83ef18
I20260928 06:27:48.801904 21782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:27:48 dist-test-slave-q03f krb5kdc[21691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:48.807569 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:48.793688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34235 (local address 127.31.214.126:38059)
0928 06:27:48.793879 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:48.793889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:48.794009 (+ 120us) server_negotiation.cc:407] Connection header received
0928 06:27:48.795501 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:48.795512 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:48.795582 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:48.795793 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:48.796570 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:48.797761 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:48.798957 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:48.799651 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:48.799772 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:48.803168 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:48.803172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:48.803173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:48.803187 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:48.805440 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:48.806011 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:48.806015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:48.806015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:48.806180 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:48.806504 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:48.806505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:48.806505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:48.806864 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:48.807058 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:48.807220 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:27:48.807335 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260928 06:27:48.808861 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:48.819797 21782 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:48.820437 21782 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:48.826427 21782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d1c1fe7bdc34f92ab54ba4bf9e22da6: Generated new TSK 0
I20260928 06:27:48.826648 21782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:48.827374 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:48.818388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54873 (local address 127.31.214.126:38059)
0928 06:27:48.818458 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:48.818461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:48.818570 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:27:48.818634 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:48.818635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:48.818647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:48.818731 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:48.819488 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:48.820282 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:48.821370 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:48.822247 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:48.822330 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:48.823991 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:48.823993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:48.823993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:48.824000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:48.825693 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:48.826298 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:48.826299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:48.826300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:48.826368 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:48.826635 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:48.826635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:48.826636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:48.826805 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:48.826890 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:48.827177 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:27:48.827254 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:27:48.830425 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 21704
W20260928 06:27:48.834738 21689 connection.cc:535] client connection to 127.31.214.126:38059 recv error: Network error: failed to read from TLS socket (remote: 127.31.214.126:38059): Connection reset by peer (error 104)
2026-09-28T06:27:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:48 dist-test-slave-q03f krb5kdc[21803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:48 dist-test-slave-q03f krb5kdc[21803](info): set up 2 sockets
Sep 28 06:27:48 dist-test-slave-q03f krb5kdc[21803](info): commencing operation
krb5kdc: starting...
W20260928 06:27:49.926192 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:50 dist-test-slave-q03f krb5kdc[21803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:50.057473 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:42071
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:42071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:50.141809 21816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:50.142125 21816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:50.142179 21816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:50.143381 21816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:50.143457 21816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:50.143481 21816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:50.143502 21816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:50.143532 21816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:50.145033 21816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:42071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:42071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:50.145323 21816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:50.145569 21816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:50.150986 21824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:50.151115 21822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:50.150990 21816 server_base.cc:1034] running on GCE node
W20260928 06:27:50.150964 21821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:50.151537 21816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:50.151824 21816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:50.152997 21816 hybrid_clock.cc:648] HybridClock initialized: now 1790576870152984 us; error 33 us; skew 500 ppm
Sep 28 06:27:50 dist-test-slave-q03f krb5kdc[21803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576870, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:50.155934 21816 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:50.156431 21816 webserver.cc:460] Webserver started at http://127.31.214.126:36713/ using document root <none> and password file <none>
I20260928 06:27:50.156716 21816 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:50.156795 21816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:50.156963 21816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:50.158205 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "831e2f1bd7d948faadddf9322ef3211d"
format_stamp: "Formatted at 2026-09-28 06:27:50 on dist-test-slave-q03f"
server_key: "00c5ce4c66b82718ed930b59f338f877"
server_key_iv: "2d27984903dc06d77b08ee46f94b6b6d"
server_key_version: "encryptionkey@0"
I20260928 06:27:50.158598 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "831e2f1bd7d948faadddf9322ef3211d"
format_stamp: "Formatted at 2026-09-28 06:27:50 on dist-test-slave-q03f"
server_key: "00c5ce4c66b82718ed930b59f338f877"
server_key_iv: "2d27984903dc06d77b08ee46f94b6b6d"
server_key_version: "encryptionkey@0"
I20260928 06:27:50.160179 21816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:27:50.161120 21831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:50.161370 21816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:50.161491 21816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "831e2f1bd7d948faadddf9322ef3211d"
format_stamp: "Formatted at 2026-09-28 06:27:50 on dist-test-slave-q03f"
server_key: "00c5ce4c66b82718ed930b59f338f877"
server_key_iv: "2d27984903dc06d77b08ee46f94b6b6d"
server_key_version: "encryptionkey@0"
I20260928 06:27:50.161571 21816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:50.183666 21816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:50.186793 21816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:50.186975 21816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:50.191154 21816 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:42071
I20260928 06:27:50.191246 21883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:42071 every 8 connection(s)
I20260928 06:27:50.191702 21816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:50.192668 21884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:50.193053 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 21816
I20260928 06:27:50.193177 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:50.193406 32601 external_mini_cluster.cc:1428] Setting key 2aefe4664c920d32c7b92173d912d25d
I20260928 06:27:50.195636 21884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d: Bootstrap starting.
I20260928 06:27:50.196405 21884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:50.196731 21884 log.cc:824] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:50.197489 21884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d: No bootstrap required, opened a new log
I20260928 06:27:50.199283 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:50.194035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46077 (local address 127.31.214.126:42071)
0928 06:27:50.194210 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:50.194218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:50.194249 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:27:50.194681 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:50.194687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:50.194738 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:50.194861 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:50.195105 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:50.196090 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:50.197234 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:50.197706 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:50.197807 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:50.198579 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:50.198583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:50.198583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:50.198756 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:50.198788 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:50.198799 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:50.198869 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:50.198992 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:27:50.199041 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 06:27:50.200026 21884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831e2f1bd7d948faadddf9322ef3211d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42071 } }
I20260928 06:27:50.200187 21884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:50.200246 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:50.200249 21884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831e2f1bd7d948faadddf9322ef3211d, State: Initialized, Role: FOLLOWER
I20260928 06:27:50.200404 21884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831e2f1bd7d948faadddf9322ef3211d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42071 } }
I20260928 06:27:50.200505 21884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:50.200551 21884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:50.200600 21884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:50.201170 21884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831e2f1bd7d948faadddf9322ef3211d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42071 } }
I20260928 06:27:50.201294 21884 leader_election.cc:302] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 831e2f1bd7d948faadddf9322ef3211d; no voters:
I20260928 06:27:50.201488 21884 leader_election.cc:288] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:50.201519 21893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:50.201648 21893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [term 1 LEADER]: Becoming Leader. State: Replica: 831e2f1bd7d948faadddf9322ef3211d, State: Running, Role: LEADER
I20260928 06:27:50.201762 21884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:50.201792 21893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831e2f1bd7d948faadddf9322ef3211d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42071 } }
I20260928 06:27:50.202310 21894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "831e2f1bd7d948faadddf9322ef3211d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831e2f1bd7d948faadddf9322ef3211d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42071 } } }
I20260928 06:27:50.202416 21894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:50.202296 21893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 831e2f1bd7d948faadddf9322ef3211d. Latest consensus state: current_term: 1 leader_uuid: "831e2f1bd7d948faadddf9322ef3211d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831e2f1bd7d948faadddf9322ef3211d" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 42071 } } }
I20260928 06:27:50.202701 21893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:50.202817 21902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:50.203341 21902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:50.205416 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:50.201694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44505 (local address 127.31.214.126:42071)
0928 06:27:50.201726 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:50.201728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:50.202061 (+ 333us) server_negotiation.cc:407] Connection header received
0928 06:27:50.202151 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:50.202152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:50.202162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:50.202207 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:50.202360 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:50.203145 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:50.204065 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:50.204326 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:50.204406 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:50.205161 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:50.205163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:50.205163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:50.205168 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:50.205183 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:50.205188 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:50.205236 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:50.205314 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:27:50.205324 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:27:50.207059 21902 catalog_manager.cc:1347] Generated new cluster ID: c6f6b3b2933c4fda9baf1ecca6f71a16
I20260928 06:27:50.207111 21902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:50.222036 21902 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:50.222854 21902 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:50.226213 21902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 831e2f1bd7d948faadddf9322ef3211d: Generated new TSK 0
I20260928 06:27:50.226438 21902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:50.234660 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 21816
W20260928 06:27:50.239526 21801 connection.cc:535] client connection to 127.31.214.126:42071 recv error: Network error: failed to read from TLS socket (remote: 127.31.214.126:42071): Connection reset by peer (error 104)
2026-09-28T06:27:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:50 dist-test-slave-q03f krb5kdc[21915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:50 dist-test-slave-q03f krb5kdc[21915](info): set up 2 sockets
Sep 28 06:27:50 dist-test-slave-q03f krb5kdc[21915](info): commencing operation
krb5kdc: starting...
W20260928 06:27:51.294301 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:51 dist-test-slave-q03f krb5kdc[21915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:51.420324 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38439
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:45217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:51.503729 21928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:51.504014 21928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:51.504053 21928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:51.505329 21928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:51.505419 21928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:51.505443 21928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:51.505465 21928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:51.505496 21928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:51.507128 21928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:45217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:38439
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:51.507440 21928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:51.507722 21928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:51.513181 21936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:51.513203 21934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:51.513259 21928 server_base.cc:1034] running on GCE node
W20260928 06:27:51.513222 21933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:51.513700 21928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:51.514038 21928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:51.515205 21928 hybrid_clock.cc:648] HybridClock initialized: now 1790576871515187 us; error 39 us; skew 500 ppm
Sep 28 06:27:51 dist-test-slave-q03f krb5kdc[21915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576871, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:51.518052 21928 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:51.518554 21928 webserver.cc:460] Webserver started at http://127.31.214.126:42647/ using document root <none> and password file <none>
I20260928 06:27:51.518797 21928 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:51.518868 21928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:51.519032 21928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:51.520169 21928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "def61fb7891b46dda2c89c87582e569c"
format_stamp: "Formatted at 2026-09-28 06:27:51 on dist-test-slave-q03f"
server_key: "1af3bb0a2881497bb92f181638d17d85"
server_key_iv: "eac36aa81b1de5e4ed5d421ef4dd2555"
server_key_version: "encryptionkey@0"
I20260928 06:27:51.520566 21928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "def61fb7891b46dda2c89c87582e569c"
format_stamp: "Formatted at 2026-09-28 06:27:51 on dist-test-slave-q03f"
server_key: "1af3bb0a2881497bb92f181638d17d85"
server_key_iv: "eac36aa81b1de5e4ed5d421ef4dd2555"
server_key_version: "encryptionkey@0"
I20260928 06:27:51.522058 21928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:27:51.522876 21943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:51.523057 21928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:51.523170 21928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "def61fb7891b46dda2c89c87582e569c"
format_stamp: "Formatted at 2026-09-28 06:27:51 on dist-test-slave-q03f"
server_key: "1af3bb0a2881497bb92f181638d17d85"
server_key_iv: "eac36aa81b1de5e4ed5d421ef4dd2555"
server_key_version: "encryptionkey@0"
I20260928 06:27:51.523249 21928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:51.527172 21928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:51.529628 21928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:51.529799 21928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:51.533998 21928 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:38439
I20260928 06:27:51.534056 21995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:38439 every 8 connection(s)
I20260928 06:27:51.534456 21928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:51.535522 21996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:51.535768 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 21928
I20260928 06:27:51.535913 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:51.536125 32601 external_mini_cluster.cc:1428] Setting key 30d9912002ab63519305323c12fb57af
I20260928 06:27:51.538712 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c: Bootstrap starting.
I20260928 06:27:51.538933 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:51.536710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44295 (local address 127.31.214.126:38439)
0928 06:27:51.536912 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:51.536920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:51.536947 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:27:51.537369 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:51.537375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:51.537402 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:51.537523 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:51.538328 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:51.538332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:51.538332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:51.538336 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:51.538375 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:51.538386 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:51.538458 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:51.538557 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:27:51.538591 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260928 06:27:51.539467 21996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:51.539816 21996 log.cc:824] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:51.539865 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:51.540550 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c: No bootstrap required, opened a new log
I20260928 06:27:51.542822 21996 raft_consensus.cc:348] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def61fb7891b46dda2c89c87582e569c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38439 } }
I20260928 06:27:51.542995 21996 raft_consensus.cc:374] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:51.543043 21996 raft_consensus.cc:729] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def61fb7891b46dda2c89c87582e569c, State: Initialized, Role: FOLLOWER
I20260928 06:27:51.543172 21996 consensus_queue.cc:260] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def61fb7891b46dda2c89c87582e569c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38439 } }
I20260928 06:27:51.543273 21996 raft_consensus.cc:388] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:51.543319 21996 raft_consensus.cc:482] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:51.543370 21996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:51.543923 21996 raft_consensus.cc:504] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def61fb7891b46dda2c89c87582e569c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38439 } }
I20260928 06:27:51.544047 21996 leader_election.cc:302] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: def61fb7891b46dda2c89c87582e569c; no voters:
I20260928 06:27:51.544237 21996 leader_election.cc:288] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:51.544293 22006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:51.544482 22006 raft_consensus.cc:686] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [term 1 LEADER]: Becoming Leader. State: Replica: def61fb7891b46dda2c89c87582e569c, State: Running, Role: LEADER
I20260928 06:27:51.544528 21996 sys_catalog.cc:564] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:51.544739 22006 consensus_queue.cc:237] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def61fb7891b46dda2c89c87582e569c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38439 } }
I20260928 06:27:51.545984 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:51.541357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50161 (local address 127.31.214.126:38439)
0928 06:27:51.541400 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:51.541402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:51.544701 (+ 3299us) server_negotiation.cc:407] Connection header received
0928 06:27:51.544829 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:51.544831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:51.544837 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:51.544909 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:51.545605 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:51.545606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:51.545606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:51.545610 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:51.545625 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:51.545630 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:51.545676 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:51.545874 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:27:51.545887 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:27:51.546036 22007 sys_catalog.cc:455] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "def61fb7891b46dda2c89c87582e569c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def61fb7891b46dda2c89c87582e569c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38439 } } }
I20260928 06:27:51.546164 22007 sys_catalog.cc:458] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:51.546401 22008 sys_catalog.cc:455] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [sys.catalog]: SysCatalogTable state changed. Reason: New leader def61fb7891b46dda2c89c87582e569c. Latest consensus state: current_term: 1 leader_uuid: "def61fb7891b46dda2c89c87582e569c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def61fb7891b46dda2c89c87582e569c" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 38439 } } }
I20260928 06:27:51.546631 22008 sys_catalog.cc:458] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:51.547003 22016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:51.547492 22016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:51.549139 22016 catalog_manager.cc:1347] Generated new cluster ID: 3f78bc6941044e9f9d7f7d43a1ad3bed
I20260928 06:27:51.549194 22016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:51.565973 22016 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:51.566205 22016 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:51.570932 22016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P def61fb7891b46dda2c89c87582e569c: Generated new TSK 0
I20260928 06:27:51.571167 22016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:51.572731 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 21928
2026-09-28T06:27:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:51 dist-test-slave-q03f krb5kdc[22027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:51 dist-test-slave-q03f krb5kdc[22027](info): set up 2 sockets
Sep 28 06:27:51 dist-test-slave-q03f krb5kdc[22027](info): commencing operation
krb5kdc: starting...
W20260928 06:27:52.614347 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:52 dist-test-slave-q03f krb5kdc[22027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:52.753731 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:41555
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:41555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:52.841584 22040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:52.841802 22040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:52.841825 22040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:52.842995 22040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:52.843040 22040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:52.843055 22040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:52.843066 22040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:52.843078 22040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:52.844723 22040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:41555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:41555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:52.845055 22040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:52.845340 22040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:52.851262 22045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:52.851248 22048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:52.851367 22046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:52.851642 22040 server_base.cc:1034] running on GCE node
I20260928 06:27:52.851948 22040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:52.852298 22040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:52.853488 22040 hybrid_clock.cc:648] HybridClock initialized: now 1790576872853476 us; error 45 us; skew 500 ppm
Sep 28 06:27:52 dist-test-slave-q03f krb5kdc[22027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576872, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:52.856412 22040 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:52.856856 22040 webserver.cc:460] Webserver started at http://127.31.214.126:40571/ using document root <none> and password file <none>
I20260928 06:27:52.857077 22040 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:52.857125 22040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:52.857226 22040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:52.858441 22040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "f1836fc1c6c9474a95067b0d038aaf42"
format_stamp: "Formatted at 2026-09-28 06:27:52 on dist-test-slave-q03f"
server_key: "cec1bd7255e3f68b5d1609a7e28a65de"
server_key_iv: "b0612934a3f099e3a7b5d7dadf244464"
server_key_version: "encryptionkey@0"
I20260928 06:27:52.858855 22040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "f1836fc1c6c9474a95067b0d038aaf42"
format_stamp: "Formatted at 2026-09-28 06:27:52 on dist-test-slave-q03f"
server_key: "cec1bd7255e3f68b5d1609a7e28a65de"
server_key_iv: "b0612934a3f099e3a7b5d7dadf244464"
server_key_version: "encryptionkey@0"
I20260928 06:27:52.860356 22040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:27:52.861181 22055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:52.861435 22040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:52.861555 22040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "f1836fc1c6c9474a95067b0d038aaf42"
format_stamp: "Formatted at 2026-09-28 06:27:52 on dist-test-slave-q03f"
server_key: "cec1bd7255e3f68b5d1609a7e28a65de"
server_key_iv: "b0612934a3f099e3a7b5d7dadf244464"
server_key_version: "encryptionkey@0"
I20260928 06:27:52.861637 22040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:52.883227 22040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:52.885929 22040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:52.886288 22040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:52.890614 22040 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:41555
I20260928 06:27:52.890727 22107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:41555 every 8 connection(s)
I20260928 06:27:52.891080 22040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:52.892103 22108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:52.894591 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42: Bootstrap starting.
I20260928 06:27:52.895150 22108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:52.895406 22108 log.cc:824] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:52.896126 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42: No bootstrap required, opened a new log
I20260928 06:27:52.897738 22108 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1836fc1c6c9474a95067b0d038aaf42" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41555 } }
I20260928 06:27:52.897898 22108 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:52.897921 22108 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1836fc1c6c9474a95067b0d038aaf42, State: Initialized, Role: FOLLOWER
I20260928 06:27:52.898139 22108 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1836fc1c6c9474a95067b0d038aaf42" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41555 } }
I20260928 06:27:52.898243 22108 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:52.898268 22108 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:52.898299 22108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:52.899192 22108 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1836fc1c6c9474a95067b0d038aaf42" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41555 } }
I20260928 06:27:52.899286 22108 leader_election.cc:302] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1836fc1c6c9474a95067b0d038aaf42; no voters:
I20260928 06:27:52.899468 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 22040
I20260928 06:27:52.899487 22108 leader_election.cc:288] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:52.899524 22111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:52.899591 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:52.899657 22111 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [term 1 LEADER]: Becoming Leader. State: Replica: f1836fc1c6c9474a95067b0d038aaf42, State: Running, Role: LEADER
I20260928 06:27:52.899755 22111 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1836fc1c6c9474a95067b0d038aaf42" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41555 } }
I20260928 06:27:52.899811 32601 external_mini_cluster.cc:1428] Setting key e4eb97587fc9dca1773c238dc8a04ff4
I20260928 06:27:52.899784 22108 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:52.901136 22112 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1836fc1c6c9474a95067b0d038aaf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1836fc1c6c9474a95067b0d038aaf42" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41555 } } }
I20260928 06:27:52.901278 22112 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:52.901502 22113 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1836fc1c6c9474a95067b0d038aaf42. Latest consensus state: current_term: 1 leader_uuid: "f1836fc1c6c9474a95067b0d038aaf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1836fc1c6c9474a95067b0d038aaf42" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41555 } } }
I20260928 06:27:52.901636 22113 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:52.901693 22125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:52.902315 22125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:52.904070 22125 catalog_manager.cc:1347] Generated new cluster ID: c4238e2d7aa44184ae3c8e6e0445330b
I20260928 06:27:52.904131 22125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:27:52 dist-test-slave-q03f krb5kdc[22027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:52.913285 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:52.900360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49001 (local address 127.31.214.126:41555)
0928 06:27:52.900564 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:52.900573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:52.900606 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:27:52.900796 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:52.900805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:52.900906 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:52.901990 (+ 1084us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:52.902109 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:52.903517 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:52.904633 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:52.905276 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:52.905380 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:52.908492 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:52.908497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:52.908498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:52.908512 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:52.911407 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:52.911848 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:52.911852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:52.911853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:52.912000 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:52.912218 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:52.912219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:52.912220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:52.912596 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:52.912783 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:52.912886 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:27:52.913042 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260928 06:27:52.913990 22125 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:52.914541 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:52.914927 22125 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:52.923929 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:52.916329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32883 (local address 127.31.214.126:41555)
0928 06:27:52.916370 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:52.916372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:52.916512 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:27:52.916573 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:52.916574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:52.916606 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:52.916663 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:52.917342 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:52.918133 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:52.919300 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:52.919606 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:52.919665 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:52.921142 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:52.921143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:52.921143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:52.921150 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:52.922761 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:52.923157 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:52.923158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:52.923158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:52.923216 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:52.923418 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:52.923419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:52.923419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:52.923539 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:52.923627 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:52.923755 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 06:27:52.923834 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:27:52.930661 22125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1836fc1c6c9474a95067b0d038aaf42: Generated new TSK 0
I20260928 06:27:52.930903 22125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:52.933666 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 22040
2026-09-28T06:27:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:52 dist-test-slave-q03f krb5kdc[22139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:52 dist-test-slave-q03f krb5kdc[22139](info): set up 2 sockets
Sep 28 06:27:52 dist-test-slave-q03f krb5kdc[22139](info): commencing operation
krb5kdc: starting...
W20260928 06:27:54.004076 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:54 dist-test-slave-q03f krb5kdc[22139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:54.133927 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35513
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:54.222229 22152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:54.222500 22152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:54.222546 22152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:54.223982 22152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:54.224074 22152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:54.224102 22152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:54.224148 22152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:54.224184 22152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:54.225908 22152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:35513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:54.226285 22152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:54.226574 22152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:54.232228 22160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:54.232237 22157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:54.232378 22152 server_base.cc:1034] running on GCE node
W20260928 06:27:54.232213 22158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:54.232769 22152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:54.233121 22152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:54.234290 22152 hybrid_clock.cc:648] HybridClock initialized: now 1790576874234273 us; error 37 us; skew 500 ppm
Sep 28 06:27:54 dist-test-slave-q03f krb5kdc[22139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576874, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:54.243801 22152 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:54.244370 22152 webserver.cc:460] Webserver started at http://127.31.214.126:38709/ using document root <none> and password file <none>
I20260928 06:27:54.244585 22152 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:54.244637 22152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:54.244746 22152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:54.245847 22152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "7ceb9512de51404fbca20f1475889554"
format_stamp: "Formatted at 2026-09-28 06:27:54 on dist-test-slave-q03f"
server_key: "3547c74ebf0798a9e9e08ada9a0a252b"
server_key_iv: "519cc7f56e3628163e8bd2f4f7d7fe49"
server_key_version: "encryptionkey@0"
I20260928 06:27:54.246268 22152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "7ceb9512de51404fbca20f1475889554"
format_stamp: "Formatted at 2026-09-28 06:27:54 on dist-test-slave-q03f"
server_key: "3547c74ebf0798a9e9e08ada9a0a252b"
server_key_iv: "519cc7f56e3628163e8bd2f4f7d7fe49"
server_key_version: "encryptionkey@0"
I20260928 06:27:54.247691 22152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:54.248521 22167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:54.248746 22152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:54.248844 22152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "7ceb9512de51404fbca20f1475889554"
format_stamp: "Formatted at 2026-09-28 06:27:54 on dist-test-slave-q03f"
server_key: "3547c74ebf0798a9e9e08ada9a0a252b"
server_key_iv: "519cc7f56e3628163e8bd2f4f7d7fe49"
server_key_version: "encryptionkey@0"
I20260928 06:27:54.248905 22152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:54.266148 22152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:54.268527 22152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:54.268646 22152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:54.272542 22152 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:35513
I20260928 06:27:54.272624 22219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:35513 every 8 connection(s)
I20260928 06:27:54.272981 22152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:54.273918 22220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:54.276335 22220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554: Bootstrap starting.
I20260928 06:27:54.276953 22220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:54.277251 22220 log.cc:824] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:54.278131 22220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554: No bootstrap required, opened a new log
I20260928 06:27:54.279842 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 22152
I20260928 06:27:54.279961 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:54.280014 22220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ceb9512de51404fbca20f1475889554" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35513 } }
I20260928 06:27:54.280170 32601 external_mini_cluster.cc:1428] Setting key 1f6ded64952db283c3caa0f0b0200f01
I20260928 06:27:54.280206 22220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:54.280249 22220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ceb9512de51404fbca20f1475889554, State: Initialized, Role: FOLLOWER
I20260928 06:27:54.280441 22220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ceb9512de51404fbca20f1475889554" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35513 } }
I20260928 06:27:54.280543 22220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:54.280668 22220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:54.280776 22220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:54.282076 22220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ceb9512de51404fbca20f1475889554" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35513 } }
I20260928 06:27:54.282193 22220 leader_election.cc:302] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ceb9512de51404fbca20f1475889554; no voters:
I20260928 06:27:54.282366 22220 leader_election.cc:288] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:54.282447 22225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:54.282614 22220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:54.282652 22225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [term 1 LEADER]: Becoming Leader. State: Replica: 7ceb9512de51404fbca20f1475889554, State: Running, Role: LEADER
I20260928 06:27:54.282867 22225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ceb9512de51404fbca20f1475889554" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35513 } }
I20260928 06:27:54.283274 22226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ceb9512de51404fbca20f1475889554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ceb9512de51404fbca20f1475889554" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35513 } } }
I20260928 06:27:54.283366 22226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:54.283594 22227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ceb9512de51404fbca20f1475889554. Latest consensus state: current_term: 1 leader_uuid: "7ceb9512de51404fbca20f1475889554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ceb9512de51404fbca20f1475889554" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 35513 } } }
I20260928 06:27:54.283907 22227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:54.284099 22234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:54.284811 22234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:54.286551 22234 catalog_manager.cc:1347] Generated new cluster ID: 777ddd56ef014a93b135af93aecb8ccd
I20260928 06:27:54.286614 22234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:54.287025 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:54.280690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48885 (local address 127.31.214.126:35513)
0928 06:27:54.280957 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:54.280968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:54.281004 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:27:54.281133 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:54.281141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:54.281217 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:54.281390 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:54.281560 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:54.283020 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:54.284269 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:54.284883 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:54.284991 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:54.286077 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:54.286080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:54.286080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:54.286328 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:54.286367 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:54.286383 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:54.286444 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:54.286743 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 06:27:54.286815 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":132,"threads_started":1}
I20260928 06:27:54.287811 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:54.294158 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:54.290300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39159 (local address 127.31.214.126:35513)
0928 06:27:54.290348 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:54.290350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:54.290565 (+ 215us) server_negotiation.cc:407] Connection header received
0928 06:27:54.290625 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:54.290626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:54.290636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:54.290699 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:54.290857 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:54.291596 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:54.292693 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:54.292989 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:54.293044 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:54.293866 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:54.293867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:54.293868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:54.293872 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:54.293889 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:54.293894 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:54.293962 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:54.294054 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 06:27:54.294080 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:27:54.309062 22234 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:54.309674 22234 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:54.316840 22234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ceb9512de51404fbca20f1475889554: Generated new TSK 0
I20260928 06:27:54.317080 22234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:54.326327 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 22152
2026-09-28T06:27:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:54 dist-test-slave-q03f krb5kdc[22251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:54 dist-test-slave-q03f krb5kdc[22251](info): set up 2 sockets
Sep 28 06:27:54 dist-test-slave-q03f krb5kdc[22251](info): commencing operation
krb5kdc: starting...
W20260928 06:27:55.378931 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:55 dist-test-slave-q03f krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:55.514412 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:34437
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:34437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:55.597538 22264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:55.597810 22264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:55.597848 22264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:55.599057 22264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:55.599151 22264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:55.599177 22264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:55.599196 22264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:55.599228 22264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:55.600871 22264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:34437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:34437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:55.601179 22264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:55.601429 22264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:55.606704 22272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:55.606704 22269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:55.606788 22270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:55.606909 22264 server_base.cc:1034] running on GCE node
I20260928 06:27:55.607208 22264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:55.607519 22264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:55.608671 22264 hybrid_clock.cc:648] HybridClock initialized: now 1790576875608635 us; error 64 us; skew 500 ppm
Sep 28 06:27:55 dist-test-slave-q03f krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576875, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:55.611492 22264 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:55.612076 22264 webserver.cc:460] Webserver started at http://127.31.214.126:35725/ using document root <none> and password file <none>
I20260928 06:27:55.612326 22264 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:55.612403 22264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:55.612564 22264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:55.613759 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "1bea2871a2c94cfab13e5e5f6f07fd83"
format_stamp: "Formatted at 2026-09-28 06:27:55 on dist-test-slave-q03f"
server_key: "315c645d611d5add0934a6ad786ba747"
server_key_iv: "bf14e5c6e60ec25444b22930d44036cc"
server_key_version: "encryptionkey@0"
I20260928 06:27:55.614176 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "1bea2871a2c94cfab13e5e5f6f07fd83"
format_stamp: "Formatted at 2026-09-28 06:27:55 on dist-test-slave-q03f"
server_key: "315c645d611d5add0934a6ad786ba747"
server_key_iv: "bf14e5c6e60ec25444b22930d44036cc"
server_key_version: "encryptionkey@0"
I20260928 06:27:55.615706 22264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:55.616747 22279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:55.616983 22264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:55.617120 22264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "1bea2871a2c94cfab13e5e5f6f07fd83"
format_stamp: "Formatted at 2026-09-28 06:27:55 on dist-test-slave-q03f"
server_key: "315c645d611d5add0934a6ad786ba747"
server_key_iv: "bf14e5c6e60ec25444b22930d44036cc"
server_key_version: "encryptionkey@0"
I20260928 06:27:55.617204 22264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:55.620846 22264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:55.623139 22264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:55.623309 22264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:55.627640 22264 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:34437
I20260928 06:27:55.627671 22331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:34437 every 8 connection(s)
I20260928 06:27:55.628021 22264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:55.629140 22332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:55.629633 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 22264
I20260928 06:27:55.629734 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:55.629982 32601 external_mini_cluster.cc:1428] Setting key 1b764e774b3770f7231e8c8752418d6d
I20260928 06:27:55.632481 22332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83: Bootstrap starting.
I20260928 06:27:55.632694 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:55.630552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49871 (local address 127.31.214.126:34437)
0928 06:27:55.630771 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:55.630782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:55.630819 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:27:55.631322 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:55.631331 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:55.631375 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:55.631558 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:55.632225 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:55.632229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:55.632230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:55.632234 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:55.632288 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:55.632304 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:55.632382 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:55.632436 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:27:55.632476 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260928 06:27:55.633200 22332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:55.633517 22332 log.cc:824] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:55.633934 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:27:55.634454 22332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83: No bootstrap required, opened a new log
I20260928 06:27:55.636850 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:55.635730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43527 (local address 127.31.214.126:34437)
0928 06:27:55.635772 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:55.635774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:55.635873 (+ 99us) server_negotiation.cc:407] Connection header received
0928 06:27:55.635917 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:55.635917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:55.635922 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:55.635978 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:55.636600 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:55.636602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:55.636602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:27:55.636608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:55.636626 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:27:55.636632 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:55.636711 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:55.636759 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:27:55.636773 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:27:55.637607 22332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bea2871a2c94cfab13e5e5f6f07fd83" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34437 } }
I20260928 06:27:55.637756 22332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:27:55.637779 22332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bea2871a2c94cfab13e5e5f6f07fd83, State: Initialized, Role: FOLLOWER
I20260928 06:27:55.637869 22332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bea2871a2c94cfab13e5e5f6f07fd83" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34437 } }
I20260928 06:27:55.637990 22332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:27:55.638036 22332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:27:55.638087 22332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:55.638672 22332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bea2871a2c94cfab13e5e5f6f07fd83" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34437 } }
I20260928 06:27:55.638762 22332 leader_election.cc:302] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bea2871a2c94cfab13e5e5f6f07fd83; no voters:
I20260928 06:27:55.638904 22332 leader_election.cc:288] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:27:55.638976 22342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:55.639176 22332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:55.639250 22342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [term 1 LEADER]: Becoming Leader. State: Replica: 1bea2871a2c94cfab13e5e5f6f07fd83, State: Running, Role: LEADER
I20260928 06:27:55.639356 22342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bea2871a2c94cfab13e5e5f6f07fd83" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34437 } }
I20260928 06:27:55.639775 22342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bea2871a2c94cfab13e5e5f6f07fd83. Latest consensus state: current_term: 1 leader_uuid: "1bea2871a2c94cfab13e5e5f6f07fd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bea2871a2c94cfab13e5e5f6f07fd83" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34437 } } }
I20260928 06:27:55.639808 22343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bea2871a2c94cfab13e5e5f6f07fd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bea2871a2c94cfab13e5e5f6f07fd83" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34437 } } }
I20260928 06:27:55.639923 22342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:55.639938 22343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:55.640141 22351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:55.640761 22351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:55.642377 22351 catalog_manager.cc:1347] Generated new cluster ID: 153da488a7e64105b76b593b14390c2e
I20260928 06:27:55.642433 22351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:55.654098 22351 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:55.654330 22351 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:55.660619 22351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bea2871a2c94cfab13e5e5f6f07fd83: Generated new TSK 0
I20260928 06:27:55.660866 22351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:27:55.669615 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 22264
2026-09-28T06:27:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1343 ms)
[----------] 18 tests from AuthTokenIssuingTest (24639 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:55 dist-test-slave-q03f krb5kdc[22363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:55 dist-test-slave-q03f krb5kdc[22363](info): set up 2 sockets
Sep 28 06:27:55 dist-test-slave-q03f krb5kdc[22363](info): commencing operation
krb5kdc: starting...
W20260928 06:27:56.743029 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:56 dist-test-slave-q03f krb5kdc[22363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:56.882115 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:41695
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:41695,127.31.214.125:38917,127.31.214.124:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:56.968070 22376 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:27:56.968331 22376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:56.968364 22376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:56.968410 22376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:56.969600 22376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:56.969671 22376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:56.969696 22376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:56.969729 22376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:56.969751 22376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:56.971292 22376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:41695,127.31.214.125:38917,127.31.214.124:32955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:41695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:56.971596 22376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:56.971838 22376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:56.977373 22384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:56.977373 22382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:56.977424 22381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:56.977808 22376 server_base.cc:1034] running on GCE node
I20260928 06:27:56.978109 22376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:56.978415 22376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:56.979614 22376 hybrid_clock.cc:648] HybridClock initialized: now 1790576876979607 us; error 66 us; skew 500 ppm
Sep 28 06:27:56 dist-test-slave-q03f krb5kdc[22363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576876, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:56.982601 22376 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:56.983144 22376 webserver.cc:460] Webserver started at http://127.31.214.126:45155/ using document root <none> and password file <none>
I20260928 06:27:56.983386 22376 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:56.983458 22376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:56.983598 22376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:56.984774 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "439c9e878a514b9f8a0ed539f746a907"
format_stamp: "Formatted at 2026-09-28 06:27:56 on dist-test-slave-q03f"
server_key: "e056be436f553afbfc21257b89a76cd5"
server_key_iv: "a6148eb946693677002824658a432d55"
server_key_version: "encryptionkey@0"
I20260928 06:27:56.985172 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "439c9e878a514b9f8a0ed539f746a907"
format_stamp: "Formatted at 2026-09-28 06:27:56 on dist-test-slave-q03f"
server_key: "e056be436f553afbfc21257b89a76cd5"
server_key_iv: "a6148eb946693677002824658a432d55"
server_key_version: "encryptionkey@0"
I20260928 06:27:56.986778 22376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:27:56.987720 22391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:56.987922 22376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:56.988054 22376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "439c9e878a514b9f8a0ed539f746a907"
format_stamp: "Formatted at 2026-09-28 06:27:56 on dist-test-slave-q03f"
server_key: "e056be436f553afbfc21257b89a76cd5"
server_key_iv: "a6148eb946693677002824658a432d55"
server_key_version: "encryptionkey@0"
I20260928 06:27:56.988140 22376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:56.997397 22376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:56.999929 22376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:57.000120 22376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:57.004653 22376 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:41695
I20260928 06:27:57.004690 22443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:41695 every 8 connection(s)
I20260928 06:27:57.005079 22376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:57.006228 22444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:57.007587 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 22376
I20260928 06:27:57.007791 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:57.007987 32601 external_mini_cluster.cc:1428] Setting key ca7c9469457f10d1d60b0f51a38d46ff
I20260928 06:27:57.007977 22444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576876, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.214.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.125@KRBTEST.COM" created.
I20260928 06:27:57.021593 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.009041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.31.214.126:41695)
0928 06:27:57.009190 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.009195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.009230 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:27:57.009792 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.009795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.009812 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.009922 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.010611 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.012012 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.013595 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.014232 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.014358 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.017493 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.017502 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.017503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.017524 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.019726 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.020303 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.020307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.020307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.020419 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.020761 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.020762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.020762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.020989 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.021058 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.021311 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.021413 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":53,"threads_started":1}
I20260928 06:27:57.021697 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.008820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:41695 (local address 127.0.0.1:58408)
0928 06:27:57.008997 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.009042 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:57.009577 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:57.009916 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:57.009919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:57.010255 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:57.010459 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.010469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.012178 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.012179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.013475 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.013479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.014466 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.014467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.014542 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.015205 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:57.015218 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:57.017408 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:57.019808 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.019816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.019819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.020099 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.020521 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.020522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.020523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.020645 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.021068 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:57.021076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:57.021162 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 06:27:57.021274 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:27:57.021498 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":47,"thread_start_us":85,"threads_started":1}
I20260928 06:27:57.022547 22444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:27:57.022965 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.022847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:38917 (local address 127.0.0.1:54804)
0928 06:27:57.022898 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.022945 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:27:57.023131 22392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
I20260928 06:27:57.023587 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.023473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:38917 (local address 127.0.0.1:54816)
0928 06:27:57.023524 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.023556 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:27:57.023703 22444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.125:38917: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab.
Entry for principal kudu/127.31.214.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab.
I20260928 06:27:57.038177 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.125:38917
--webserver_interface=127.31.214.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab
--principal=kudu/127.31.214.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:41695,127.31.214.125:38917,127.31.214.124:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:27:57.072947 22444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } attempt: 1
I20260928 06:27:57.073606 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.073450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:38917 (local address 127.0.0.1:54820)
0928 06:27:57.073517 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.073568 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:27:57.074030 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.073912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:38917 (local address 127.0.0.1:54830)
0928 06:27:57.073981 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.074011 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 06:27:57.074198 22444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.125:38917: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
W20260928 06:27:57.123951 22451 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:27:57.124193 22451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:57.124212 22451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:57.124235 22451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:57.125473 22451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:57.125517 22451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:57.125532 22451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:57.125545 22451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:57.125557 22451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:57.127122 22451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:41695,127.31.214.125:38917,127.31.214.124:32955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab
--principal=kudu/127.31.214.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.125:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.214.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:57.127395 22451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:57.127652 22451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:57.133375 22457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:57.133380 22456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:57.133370 22459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:57.133693 22451 server_base.cc:1034] running on GCE node
I20260928 06:27:57.134078 22451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:57.134377 22451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:57.135704 22451 hybrid_clock.cc:648] HybridClock initialized: now 1790576877135680 us; error 46 us; skew 500 ppm
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:57.138582 22451 init.cc:377] Logged in from keytab as kudu/127.31.214.125@KRBTEST.COM (short username kudu)
I20260928 06:27:57.139132 22451 webserver.cc:460] Webserver started at http://127.31.214.125:37745/ using document root <none> and password file <none>
I20260928 06:27:57.139398 22451 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:57.139477 22451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:57.139634 22451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:57.140796 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/data/instance:
uuid: "f809a790bd4448959b5c50c35fa37a22"
format_stamp: "Formatted at 2026-09-28 06:27:57 on dist-test-slave-q03f"
server_key: "c2a9141b4be71171ad429feb4244b44b"
server_key_iv: "4e8b9965d83d3684ad90a94ce1490172"
server_key_version: "encryptionkey@0"
I20260928 06:27:57.141219 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/wal/instance:
uuid: "f809a790bd4448959b5c50c35fa37a22"
format_stamp: "Formatted at 2026-09-28 06:27:57 on dist-test-slave-q03f"
server_key: "c2a9141b4be71171ad429feb4244b44b"
server_key_iv: "4e8b9965d83d3684ad90a94ce1490172"
server_key_version: "encryptionkey@0"
I20260928 06:27:57.142341 22444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } attempt: 2
I20260928 06:27:57.142771 22451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:27:57.142838 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.142673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:38917 (local address 127.0.0.1:54838)
0928 06:27:57.142757 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.142801 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 06:27:57.143304 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.143174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:38917 (local address 127.0.0.1:54854)
0928 06:27:57.143236 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.143271 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:27:57.143435 22444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.125:38917: Network error: Client connection negotiation failed: client connection to 127.31.214.125:38917: connect: Connection refused (error 111)
I20260928 06:27:57.143827 22466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:57.144101 22451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:27:57.144217 22451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/wal
uuid: "f809a790bd4448959b5c50c35fa37a22"
format_stamp: "Formatted at 2026-09-28 06:27:57 on dist-test-slave-q03f"
server_key: "c2a9141b4be71171ad429feb4244b44b"
server_key_iv: "4e8b9965d83d3684ad90a94ce1490172"
server_key_version: "encryptionkey@0"
I20260928 06:27:57.144312 22451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:57.157930 22451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:57.160547 22451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:57.160724 22451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:57.165113 22451 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.125:38917
I20260928 06:27:57.165201 22518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.125:38917 every 8 connection(s)
I20260928 06:27:57.165618 22451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
I20260928 06:27:57.166769 22519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:57.168361 22519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:27:57.173839 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 22451
I20260928 06:27:57.173936 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-1/wal/instance
I20260928 06:27:57.174211 32601 external_mini_cluster.cc:1428] Setting key e8833e3161cd3b5b8768b5c1686e9e61
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:57.179826 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.169112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.31.214.126:41695)
0928 06:27:57.169218 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.169221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.169405 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:27:57.170088 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.170090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.170103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.170173 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.170899 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.171491 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.172617 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.172875 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.172997 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.176116 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.176118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.176118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.176129 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.177699 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.178291 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.178292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.178293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.178356 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.178798 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.178800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.178800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.178946 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.179012 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.179510 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.179663 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:27:57.179867 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.169030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:41695 (local address 127.0.0.1:58414)
0928 06:27:57.169254 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.169304 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:57.169890 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:57.170267 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:57.170270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:57.170553 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:57.170780 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.170785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.171601 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.171602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.172512 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.172515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.173018 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.173019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.173172 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.173809 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:57.173818 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:57.175942 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:57.177798 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.177806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.177809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.178205 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.178437 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.178438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.178439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.178655 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.179069 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:57.179079 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:57.179297 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 06:27:57.179423 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 06:27:57.179633 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
I20260928 06:27:57.180629 22519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.214.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.124@KRBTEST.COM" created.
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:27:57.191041 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.180958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:38917 (local address 127.0.0.1:54856)
0928 06:27:57.181014 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.181025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:57.181063 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:57.182045 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:57.182046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:57.182203 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:57.182255 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.182256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.183260 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.183261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.183863 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.183866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.184360 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.184361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.184431 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.184946 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:57.184952 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:57.187693 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:57.189762 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.189763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.189764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.190075 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.190314 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.190315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.190315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.190373 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.190760 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:57.190761 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:57.190802 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:27:57.190856 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:27:57.190940 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:27:57.191202 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.181069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.31.214.125:38917)
0928 06:27:57.181850 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.181856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.181885 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:27:57.181910 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.181911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.181923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.182045 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.182342 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.183142 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.183952 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.184279 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.184365 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.187793 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.187807 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.187808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.187816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.189662 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.190161 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.190165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.190166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.190236 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.190438 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.190439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.190440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.190689 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.190777 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.190920 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.191018 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":107,"threads_started":1}
I20260928 06:27:57.191900 22519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:27:57.192304 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.192144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43272)
0928 06:27:57.192199 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.192270 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:27:57.192566 22470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
I20260928 06:27:57.193070 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.192948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43286)
0928 06:27:57.193003 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.193038 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:27:57.193215 22519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:32955: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab.
Entry for principal kudu/127.31.214.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab.
I20260928 06:27:57.205580 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.124:32955
--webserver_interface=127.31.214.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab
--principal=kudu/127.31.214.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:41695,127.31.214.125:38917,127.31.214.124:32955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:27:57.234730 22444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } attempt: 3
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576876, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:27:57.242506 22519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } attempt: 1
I20260928 06:27:57.243093 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.242940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43288)
0928 06:27:57.242982 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.243053 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 06:27:57.243276 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.235377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.31.214.125:38917)
0928 06:27:57.235449 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.235452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.235465 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:27:57.235574 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.235575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.235590 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.235672 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.236084 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.236861 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.237610 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.237993 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.238078 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.240293 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.240294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.240295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.240302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.241868 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.242315 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.242316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.242317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.242378 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.242684 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.242686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.242686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.242848 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.242922 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.243105 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.243200 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:27:57.243446 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.243362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43290)
0928 06:27:57.243397 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.243431 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 06:27:57.243562 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.235234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:38917 (local address 127.0.0.1:54860)
0928 06:27:57.235329 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.235346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:57.235407 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:57.235743 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:57.235745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:57.235909 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:57.235985 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.235987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.236970 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.236972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.237515 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.237517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.238080 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.238081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.238146 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.238741 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:57.238746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:57.240185 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:57.241980 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.241982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.241983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.242221 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.242525 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.242526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.242526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.242598 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.242913 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:57.242914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:57.242969 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 06:27:57.243058 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:27:57.243266 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 06:27:57.243613 22519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:32955: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
I20260928 06:27:57.243961 22444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:27:57.244271 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.244187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43302)
0928 06:27:57.244219 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.244249 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 06:27:57.244580 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.244525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43318)
0928 06:27:57.244547 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.244567 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 06:27:57.244683 22444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:32955: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
I20260928 06:27:57.275861 22444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } attempt: 1
I20260928 06:27:57.276451 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.276296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43324)
0928 06:27:57.276374 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.276418 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:27:57.277033 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.276896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43340)
0928 06:27:57.276965 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.277001 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 06:27:57.277139 22444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:32955: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
W20260928 06:27:57.295230 22526 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:27:57.295512 22526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:57.295557 22526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:57.295603 22526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:57.296816 22526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:57.296891 22526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:57.296914 22526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:57.296934 22526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:57.296967 22526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:57.298599 22526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:41695,127.31.214.125:38917,127.31.214.124:32955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab
--principal=kudu/127.31.214.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.124:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.214.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:57.298914 22526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:57.299268 22526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:57.305361 22534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:57.305377 22531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:57.305483 22526 server_base.cc:1034] running on GCE node
W20260928 06:27:57.305351 22532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:57.306100 22526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:57.306420 22526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:57.307644 22526 hybrid_clock.cc:648] HybridClock initialized: now 1790576877307625 us; error 33 us; skew 500 ppm
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:57.310685 22526 init.cc:377] Logged in from keytab as kudu/127.31.214.124@KRBTEST.COM (short username kudu)
I20260928 06:27:57.311138 22526 webserver.cc:460] Webserver started at http://127.31.214.124:39525/ using document root <none> and password file <none>
I20260928 06:27:57.311396 22526 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:57.311445 22526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:57.311549 22526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:57.311757 22519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } attempt: 2
I20260928 06:27:57.312178 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.312036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43348)
0928 06:27:57.312104 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.312146 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:27:57.312572 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.312479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43354)
0928 06:27:57.312521 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.312551 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:27:57.312691 22519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:32955: Network error: Client connection negotiation failed: client connection to 127.31.214.124:32955: connect: Connection refused (error 111)
I20260928 06:27:57.313030 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/data/instance:
uuid: "de67795112b7457aa3858f6cbca8559e"
format_stamp: "Formatted at 2026-09-28 06:27:57 on dist-test-slave-q03f"
server_key: "0f8cea5fee6a5805ff359710e71de669"
server_key_iv: "55f4fb2e2de5388aa29e0bc608f23fd1"
server_key_version: "encryptionkey@0"
I20260928 06:27:57.313407 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/wal/instance:
uuid: "de67795112b7457aa3858f6cbca8559e"
format_stamp: "Formatted at 2026-09-28 06:27:57 on dist-test-slave-q03f"
server_key: "0f8cea5fee6a5805ff359710e71de669"
server_key_iv: "55f4fb2e2de5388aa29e0bc608f23fd1"
server_key_version: "encryptionkey@0"
I20260928 06:27:57.315019 22526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 06:27:57.316031 22541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:57.316370 22526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:57.316511 22526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/wal
uuid: "de67795112b7457aa3858f6cbca8559e"
format_stamp: "Formatted at 2026-09-28 06:27:57 on dist-test-slave-q03f"
server_key: "0f8cea5fee6a5805ff359710e71de669"
server_key_iv: "55f4fb2e2de5388aa29e0bc608f23fd1"
server_key_version: "encryptionkey@0"
I20260928 06:27:57.316689 22526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:57.327725 22526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:57.330437 22526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:57.330572 22526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:57.335522 22526 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.124:32955
I20260928 06:27:57.335575 22593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.124:32955 every 8 connection(s)
I20260928 06:27:57.336014 22526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
I20260928 06:27:57.337075 22594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:57.338866 22594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:27:57.341159 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 22526
I20260928 06:27:57.341251 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790576570121568-32601-0/minicluster-data/master-2/wal/instance
I20260928 06:27:57.341465 32601 external_mini_cluster.cc:1428] Setting key 25a6c075c440722fd51fbd3acd37cc43
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
I20260928 06:27:57.350836 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.339731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.31.214.126:41695)
0928 06:27:57.339841 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.339844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.340064 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:27:57.340615 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.340617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.340629 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.340677 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.341392 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.342070 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.343711 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.344020 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.344105 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.346862 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.346864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.346864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.346872 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.348836 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.349401 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.349403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.349403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.349596 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.349908 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.349909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.349909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.350087 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.350146 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.350634 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.350723 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:27:57.350976 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.339677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:41695 (local address 127.0.0.1:58428)
0928 06:27:57.339900 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.339954 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:57.340490 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:57.340746 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:57.340750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:57.341047 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:57.341296 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.341304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.342157 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.342159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.343615 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.343618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.344147 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.344148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.344264 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.344898 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:57.344909 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:57.346766 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:57.348972 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.348981 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.348983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.349298 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.349694 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.349695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.349695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.349821 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.350164 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:57.350172 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:57.350369 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 06:27:57.350582 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 06:27:57.350738 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":105,"threads_started":1}
I20260928 06:27:57.351756 22594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:27:57.352294 22444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } attempt: 2
I20260928 06:27:57.353118 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.341920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.31.214.124:32955)
0928 06:27:57.342115 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.342119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.342280 (+ 161us) server_negotiation.cc:407] Connection header received
0928 06:27:57.342345 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.342347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.342365 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.342425 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.343282 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.344250 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.345402 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.345742 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.345858 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.348974 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.348982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.348982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.348990 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.351308 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.351841 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.351845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.351845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.351932 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.352136 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.352137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.352138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.352327 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.352461 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.352922 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.353002 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":122,"threads_started":1}
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576876, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:57.361354 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.352228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.31.214.125:38917)
0928 06:27:57.352319 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.352321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.352351 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:27:57.352442 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.352444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.352456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.352490 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.352885 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.353620 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.354658 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.355065 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.355166 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.357919 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.357921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.357922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.357979 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.360036 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.360518 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.360520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.360520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.360585 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.360795 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.360796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.360796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.360945 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.360979 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.361168 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.361246 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 06:27:57.361369 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.352161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:38917 (local address 127.0.0.1:54876)
0928 06:27:57.352299 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.352311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:57.352364 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:57.352571 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:57.352572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:57.352723 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:57.352791 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.352792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.353702 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.353703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.354564 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.354566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.355163 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.355164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.355255 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.355755 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:57.355761 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:57.357808 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:57.360151 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.360152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.360153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.360426 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.360663 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.360664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.360664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.360723 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.361026 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:57.361027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:57.361064 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:27:57.361140 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:27:57.361244 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 06:27:57.361740 22594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:27:57.362255 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.352548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43378)
0928 06:27:57.352601 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.352613 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:57.352654 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:57.352972 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:57.352973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:57.353187 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:57.353246 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.353248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.354334 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.354335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.354949 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.354951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.355490 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.355491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.355573 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.355928 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:57.355933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:57.357891 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:57.360898 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.360900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.360901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.361184 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.361471 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.361472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.361472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.361549 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.361845 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:57.361846 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:57.361897 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:27:57.362038 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 06:27:57.362152 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:27:57.362339 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.352614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.31.214.124:32955)
0928 06:27:57.352804 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.352806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.352813 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:27:57.352838 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.352839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.352852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.353335 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.353382 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.354262 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.355068 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.355389 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.355476 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.358033 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.358034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.358035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.358042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.360813 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.361286 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.361288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.361288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.361379 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.361638 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.361638 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.361639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.361770 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.361837 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.362082 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.362176 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":121,"threads_started":1}
I20260928 06:27:57.363655 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.354090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.31.214.126:41695)
0928 06:27:57.354138 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.354140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.354197 (+ 57us) server_negotiation.cc:407] Connection header received
0928 06:27:57.354261 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.354262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.354274 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.354331 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.355005 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.355656 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.356641 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.356847 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.356921 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.359717 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.359718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.359719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.359727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.362113 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.362771 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.362772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.362772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.362833 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.363052 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.363052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.363053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.363186 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.363365 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.363474 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.363562 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:27:57.364636 22444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907: Bootstrap starting.
I20260928 06:27:57.365492 22444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:57.365898 22444 log.cc:824] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
I20260928 06:27:57.366868 22444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907: No bootstrap required, opened a new log
I20260928 06:27:57.368815 22444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } }
I20260928 06:27:57.368975 22444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439c9e878a514b9f8a0ed539f746a907, State: Initialized, Role: FOLLOWER
I20260928 06:27:57.369086 22444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } }
I20260928 06:27:57.369478 22444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:57.369462 22601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.369688 22601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:27:57.369910 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.362106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43386)
0928 06:27:57.362429 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.362440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:57.362486 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:57.362627 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:57.362627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:57.362772 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:57.362811 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.362813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.363747 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.363748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.364298 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.364300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.364695 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.364696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.364746 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.365214 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:57.365218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:57.366750 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:57.368745 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.368746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.368747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.368961 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.369173 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.369174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.369174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.369251 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.369602 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:57.369603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:57.369639 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:27:57.369689 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:27:57.369800 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307}
I20260928 06:27:57.370290 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.362197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.31.214.124:32955)
0928 06:27:57.362239 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.362241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.362534 (+ 293us) server_negotiation.cc:407] Connection header received
0928 06:27:57.362549 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.362550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.362559 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.362596 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.362888 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.363658 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.364385 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.364609 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.364696 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.366840 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.366841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.366841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.366847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.368652 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.369061 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.369062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.369062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.369114 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.369330 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.369331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.369332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.369517 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.370052 (+ 535us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.370100 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.370187 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:27:57.371739 22612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:27:57.371845 22612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:27:57.372045 22594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e: Bootstrap starting.
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:27:57.372884 22594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:57.373256 22594 log.cc:824] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:57.374408 22594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e: No bootstrap required, opened a new log
I20260928 06:27:57.372442 22376 master_runner.cc:186] Error getting master registration for 127.31.214.125:38917: OK, instance_id { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" instance_seqno: 1790576877163830 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:27:57.374977 22376 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:27:57.376925 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.366985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.31.214.125:38917)
0928 06:27:57.367108 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.367111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.367119 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:27:57.367145 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.367146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.367156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.367218 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.367818 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.368704 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.369744 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.370141 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.370190 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.373358 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.373359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.373360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.373368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.375393 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.375833 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.375834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.375835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.375909 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.376151 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.376152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.376152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.376309 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.376486 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.376740 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.376828 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 06:27:57.376953 22594 raft_consensus.cc:348] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } }
I20260928 06:27:57.377300 22594 raft_consensus.cc:729] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de67795112b7457aa3858f6cbca8559e, State: Initialized, Role: FOLLOWER
I20260928 06:27:57.377519 22594 consensus_queue.cc:260] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } }
I20260928 06:27:57.377920 22594 sys_catalog.cc:564] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:57.378094 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.378201 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:27:57.378911 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 06:27:57.380023 22626 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:27:57.380084 22626 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:27:57.380034 22418 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "439c9e878a514b9f8a0ed539f746a907"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58442
I20260928 06:27:57.380218 22418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:27:57.380256 22418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:57.381140 22418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } }
I20260928 06:27:57.381414 22418 leader_election.cc:288] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [CANDIDATE]: Term 1 election: Requested vote from peers f809a790bd4448959b5c50c35fa37a22 (127.31.214.125:38917), de67795112b7457aa3858f6cbca8559e (127.31.214.124:32955)
I20260928 06:27:57.381922 22568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "439c9e878a514b9f8a0ed539f746a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "de67795112b7457aa3858f6cbca8559e"
I20260928 06:27:57.382099 22568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:57.382984 22568 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439c9e878a514b9f8a0ed539f746a907 in term 1.
I20260928 06:27:57.383185 22395 leader_election.cc:302] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 439c9e878a514b9f8a0ed539f746a907, de67795112b7457aa3858f6cbca8559e; no voters:
I20260928 06:27:57.383311 22601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:27:57.383466 22601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [term 1 LEADER]: Becoming Leader. State: Replica: 439c9e878a514b9f8a0ed539f746a907, State: Running, Role: LEADER
I20260928 06:27:57.383561 22601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } }
I20260928 06:27:57.384021 22601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439c9e878a514b9f8a0ed539f746a907. Latest consensus state: current_term: 1 leader_uuid: "439c9e878a514b9f8a0ed539f746a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.384110 22601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:57.384920 22526 master_runner.cc:186] Error getting master registration for 127.31.214.125:38917: OK, instance_id { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" instance_seqno: 1790576877163830 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:27:57.387681 22526 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:27:57.387907 22637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:27:57.388283 22637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:27:57.396119 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.384278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.31.214.125:38917)
0928 06:27:57.387380 (+ 3102us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.387383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.387395 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:27:57.387419 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.387507 (+ 88us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.387520 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.387609 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.388452 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.389250 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.390392 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.390741 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.390794 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.392548 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.392550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.392550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.392559 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.394605 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.395078 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.395080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.395080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.395141 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.395379 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.395380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.395380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.395540 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.395640 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.395889 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.395979 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3070}
I20260928 06:27:57.396869 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.383044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.31.214.126:41695)
0928 06:27:57.383106 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.383109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.383687 (+ 578us) server_negotiation.cc:407] Connection header received
0928 06:27:57.383775 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.383776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.383789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.383858 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.384745 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.385585 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.386741 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.387091 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.387173 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.392112 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.392113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.392114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.392124 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.394782 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.395480 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.395481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.395482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.395538 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.395906 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.395907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.395907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.396051 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.396144 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.396679 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.396770 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:27:57.397750 22568 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [term 1 FOLLOWER]: Refusing update from remote peer 439c9e878a514b9f8a0ed539f746a907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:57.398180 22635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:57.400028 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439c9e878a514b9f8a0ed539f746a907. Latest consensus state: current_term: 1 leader_uuid: "439c9e878a514b9f8a0ed539f746a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.400168 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:27:57.403965 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "439c9e878a514b9f8a0ed539f746a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.404109 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:57.404115 22519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } attempt: 3
I20260928 06:27:57.404712 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "439c9e878a514b9f8a0ed539f746a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.404814 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:27:57.405097 22639 mvcc.cc:205] Tried to move back new op lower bound from 7334202889817292800 to 7334202889763979264. Current Snapshot: MvccSnapshot[applied={T|T < 7334202889817292800}]
I20260928 06:27:57.405200 22637 catalog_manager.cc:1347] Generated new cluster ID: 342eefa64ea8465eac3b1ba225748aef
I20260928 06:27:57.405263 22637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:27:57.405211 22638 mvcc.cc:205] Tried to move back new op lower bound from 7334202889817292800 to 7334202889763979264. Current Snapshot: MvccSnapshot[applied={T|T < 7334202889817292800 or (T in {7334202889817292800})}]
Sep 28 06:27:57 dist-test-slave-q03f krb5kdc[22363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576877, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
I20260928 06:27:57.411161 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.383396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.31.214.124:32955)
0928 06:27:57.383439 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.383441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.386803 (+ 3362us) server_negotiation.cc:407] Connection header received
0928 06:27:57.386898 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.386899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.386909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.386940 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.398495 (+ 11555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.399174 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.400863 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.401103 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.401198 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.403019 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.403020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.403021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.403029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.407987 (+ 4958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.410062 (+ 2075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.410064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.410064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.410137 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.410368 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.410368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.410369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.410542 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.410682 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.410935 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.411036 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:27:57.414639 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.404405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:32955 (local address 127.0.0.1:43400)
0928 06:27:57.404507 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:57.404521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:57.404579 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:57.404864 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:57.404866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:57.405054 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:57.405109 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.405111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.406129 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.406130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.407005 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:57.407007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.408530 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.408531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:57.408611 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.409030 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:57.409035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:57.410913 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:57.413184 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.413185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.413186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.413425 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.413757 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:57.413758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:57.413759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:57.413826 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:57.414162 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:57.414163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:57.414241 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 06:27:57.414310 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:27:57.414424 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 06:27:57.415071 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:57.404663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.31.214.124:32955)
0928 06:27:57.404710 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.404712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.404719 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:27:57.404741 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.404742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.404754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.404807 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.405207 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.406029 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.408058 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:57.408413 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:57.408472 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:57.411030 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:57.411031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:57.411031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:57.411039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:57.413074 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.413505 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.413506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.413506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.413673 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:57.413894 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:57.413895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:57.413895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:57.414066 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:57.414131 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:57.414867 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 06:27:57.414970 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:27:57.416863 22519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22: Bootstrap starting.
I20260928 06:27:57.417759 22519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22: Neither blocks nor log segments found. Creating new log.
I20260928 06:27:57.418149 22519 log.cc:824] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22: Log is configured to *not* fsync() on all Append() calls
I20260928 06:27:57.420053 22519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22: No bootstrap required, opened a new log
I20260928 06:27:57.422415 22519 raft_consensus.cc:348] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } }
I20260928 06:27:57.422580 22519 raft_consensus.cc:729] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f809a790bd4448959b5c50c35fa37a22, State: Initialized, Role: FOLLOWER
I20260928 06:27:57.422677 22519 consensus_queue.cc:260] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } }
I20260928 06:27:57.423076 22519 sys_catalog.cc:564] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:27:57.423156 22492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:27:57.424299 22492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [term 1 FOLLOWER]: Refusing update from remote peer 439c9e878a514b9f8a0ed539f746a907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:27:57.424535 22493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "439c9e878a514b9f8a0ed539f746a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f809a790bd4448959b5c50c35fa37a22"
I20260928 06:27:57.425909 22601 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:27:57.426430 22493 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 439c9e878a514b9f8a0ed539f746a907 in term 1.
I20260928 06:27:57.429005 22656 sys_catalog.cc:455] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439c9e878a514b9f8a0ed539f746a907. Latest consensus state: current_term: 1 leader_uuid: "439c9e878a514b9f8a0ed539f746a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.429121 22656 sys_catalog.cc:458] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:27:57.429221 22659 sys_catalog.cc:455] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "439c9e878a514b9f8a0ed539f746a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.429281 22659 sys_catalog.cc:458] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:27:57.434059 22644 sys_catalog.cc:455] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "439c9e878a514b9f8a0ed539f746a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.435015 22644 sys_catalog.cc:458] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:27:57.435675 22655 catalog_manager.cc:1259] Loaded cluster ID: 342eefa64ea8465eac3b1ba225748aef
I20260928 06:27:57.435724 22655 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22: loading cluster ID for follower catalog manager: success
W20260928 06:27:57.435823 22655 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260928 06:27:57.435859 22655 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:27:57.480358 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "439c9e878a514b9f8a0ed539f746a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "439c9e878a514b9f8a0ed539f746a907" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 41695 } } peers { permanent_uuid: "f809a790bd4448959b5c50c35fa37a22" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 38917 } } peers { permanent_uuid: "de67795112b7457aa3858f6cbca8559e" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 32955 } } }
I20260928 06:27:57.480508 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907 [sys.catalog]: This master's current role is: LEADER
I20260928 06:27:57.482640 22637 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:27:57.483654 22637 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:27:57.489748 22637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 439c9e878a514b9f8a0ed539f746a907: Generated new TSK 0
I20260928 06:27:57.490145 22637 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:27:57.518549 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.516488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.31.214.125:38917)
0928 06:27:57.516542 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.516545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.517716 (+ 1171us) server_negotiation.cc:407] Connection header received
0928 06:27:57.517853 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.517854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.517866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.517899 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.518471 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54916: BlockingRecv error: recv got EOF from 127.0.0.1:54916 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 06:27:57.519573 22447 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.516816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.31.214.126:41695)
0928 06:27:57.516864 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.516866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.518994 (+ 2128us) server_negotiation.cc:407] Connection header received
0928 06:27:57.519113 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.519114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.519128 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.519159 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.519501 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58470: BlockingRecv error: recv got EOF from 127.0.0.1:58470 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:27:57.520563 22599 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.517334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.31.214.124:32955)
0928 06:27:57.517387 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.517389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.520023 (+ 2634us) server_negotiation.cc:407] Connection header received
0928 06:27:57.520138 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.520139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.520150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.520181 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.520503 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43404: BlockingRecv error: recv got EOF from 127.0.0.1:43404 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 06:27:57.523236 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.522453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.31.214.125:38917)
0928 06:27:57.522492 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.522493 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.522675 (+ 182us) server_negotiation.cc:407] Connection header received
0928 06:27:57.522787 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.522788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.522811 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.522842 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.523183 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54924: BlockingRecv error: recv got EOF from 127.0.0.1:54924 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:27:57.526473 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.525641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.31.214.125:38917)
0928 06:27:57.525676 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.525678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.525895 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:27:57.526031 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.526032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.526043 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.526083 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.526423 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54930: BlockingRecv error: recv got EOF from 127.0.0.1:54930 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:27:57.530705 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.529883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.31.214.125:38917)
0928 06:27:57.529917 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.529918 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.530206 (+ 288us) server_negotiation.cc:407] Connection header received
0928 06:27:57.530304 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.530305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.530315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.530342 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.530652 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54946: BlockingRecv error: recv got EOF from 127.0.0.1:54946 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 06:27:57.537001 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.536131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.31.214.125:38917)
0928 06:27:57.536176 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.536178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.536418 (+ 240us) server_negotiation.cc:407] Connection header received
0928 06:27:57.536522 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.536523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.536535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.536566 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.536942 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54954: BlockingRecv error: recv got EOF from 127.0.0.1:54954 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:27:57.547770 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.546660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.31.214.125:38917)
0928 06:27:57.546707 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.546709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.547036 (+ 327us) server_negotiation.cc:407] Connection header received
0928 06:27:57.547167 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.547168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.547181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.547212 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.547686 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54962: BlockingRecv error: recv got EOF from 127.0.0.1:54962 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:27:57.566639 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.565581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54978 (local address 127.31.214.125:38917)
0928 06:27:57.565629 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.565631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.565917 (+ 286us) server_negotiation.cc:407] Connection header received
0928 06:27:57.566102 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.566103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.566118 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.566152 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.566576 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54978: BlockingRecv error: recv got EOF from 127.0.0.1:54978 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:27:57.601205 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.600353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.31.214.125:38917)
0928 06:27:57.600406 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.600409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.600538 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:27:57.600592 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.600593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.600604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.600661 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.601115 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54982: BlockingRecv error: recv got EOF from 127.0.0.1:54982 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 06:27:57.667338 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.666466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.31.214.125:38917)
0928 06:27:57.666555 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.666559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.666675 (+ 116us) server_negotiation.cc:407] Connection header received
0928 06:27:57.666744 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.666745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.666756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.666816 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.667256 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54990: BlockingRecv error: recv got EOF from 127.0.0.1:54990 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 06:27:57.797520 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:57.796584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.31.214.125:38917)
0928 06:27:57.796657 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:57.796660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:57.796832 (+ 172us) server_negotiation.cc:407] Connection header received
0928 06:27:57.796868 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:57.796869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:57.796882 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:57.796943 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:57.797430 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55004: BlockingRecv error: recv got EOF from 127.0.0.1:55004 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 06:27:58.055650 22523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:27:58.054923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.31.214.125:38917)
0928 06:27:58.055001 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:58.055003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:58.055014 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:27:58.055043 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:58.055044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:58.055054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:58.055114 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:58.055548 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55014: BlockingRecv error: recv got EOF from 127.0.0.1:55014 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:27:58.380570 22626 catalog_manager.cc:1259] Loaded cluster ID: 342eefa64ea8465eac3b1ba225748aef
I20260928 06:27:58.380648 22626 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e: loading cluster ID for follower catalog manager: success
I20260928 06:27:58.381675 22626 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e: acquiring CA information for follower catalog manager: success
I20260928 06:27:58.382109 22626 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de67795112b7457aa3858f6cbca8559e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:27:58.437106 22655 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22: acquiring CA information for follower catalog manager: success
I20260928 06:27:58.437525 22655 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f809a790bd4448959b5c50c35fa37a22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:27:58.572522 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:58.569063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.31.214.125:38917)
0928 06:27:58.569204 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:58.569206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:58.569313 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:27:58.569434 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:58.569435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:58.569586 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:58.569694 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:58.569884 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.570681 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.571530 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.571754 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.571824 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:58.572022 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:58.572140 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:58.572215 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:58.572329 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:27:58.572360 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260928 06:27:58.577725 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:58.574998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.31.214.124:32955)
0928 06:27:58.575222 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:58.575224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:58.575233 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:27:58.575274 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:58.575275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:58.575314 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:58.575393 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:58.575506 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.576027 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.576825 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.577058 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.577125 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:58.577340 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:58.577461 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:58.577552 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:58.577629 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:27:58.577654 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":167,"threads_started":1}
I20260928 06:27:58.583438 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:58.580205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.31.214.125:38917)
0928 06:27:58.580276 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:58.580279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:58.580431 (+ 152us) server_negotiation.cc:407] Connection header received
0928 06:27:58.580514 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:58.580515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:58.580523 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:58.580588 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:58.580812 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.581592 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.582613 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.582911 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.582979 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:58.583164 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:58.583203 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:58.583256 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:58.583311 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 06:27:58.583334 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:27:58.583570 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:58.580364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.31.214.124:32955)
0928 06:27:58.580434 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:58.580436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:58.580609 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:27:58.580697 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:58.580699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:58.580711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:58.580768 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:58.580928 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.581456 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.582709 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.583001 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.583080 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:58.583230 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:58.583272 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:58.583329 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:58.583474 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:27:58.583502 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:27:58.584072 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:58.580070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.31.214.126:41695)
0928 06:27:58.580292 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:58.580294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:58.580305 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:27:58.580495 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:58.580496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:58.580557 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:58.580651 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:27:58.580873 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.581654 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.582813 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:58.583151 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:58.583218 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:58.583605 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:27:58.583736 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:27:58.583827 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:58.583927 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:27:58.583956 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":71,"threads_started":1}
I20260928 06:27:58.586364 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 22376
I20260928 06:27:58.592607 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 22451
I20260928 06:27:58.598213 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 22526
2026-09-28T06:27:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2930 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:27:58 dist-test-slave-q03f krb5kdc[22744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:27:58 dist-test-slave-q03f krb5kdc[22744](info): set up 2 sockets
Sep 28 06:27:58 dist-test-slave-q03f krb5kdc[22744](info): commencing operation
krb5kdc: starting...
W20260928 06:27:59.682564 32601 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/test-user.keytab.
Sep 28 06:27:59 dist-test-slave-q03f krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:27:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.214.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
Entry for principal kudu/127.31.214.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab.
I20260928 06:27:59.818068 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:34711
--webserver_interface=127.31.214.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:34711,127.31.214.125:42339,127.31.214.124:38913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:27:59.904786 22757 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:27:59.905009 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:27:59.905027 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:27:59.905049 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:27:59.906256 22757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:27:59.906304 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:27:59.906319 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:27:59.906332 22757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:27:59.906343 22757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:27:59.907859 22757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:34711,127.31.214.125:42339,127.31.214.124:38913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.126.keytab
--principal=kudu/127.31.214.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.126:34711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.214.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:27:59.908075 22757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:27:59.908257 22757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:27:59.913619 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:27:59.913604 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:59.913738 22757 server_base.cc:1034] running on GCE node
W20260928 06:27:59.913620 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:27:59.914183 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:27:59.914466 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:27:59.915688 22757 hybrid_clock.cc:648] HybridClock initialized: now 1790576879915652 us; error 46 us; skew 500 ppm
Sep 28 06:27:59 dist-test-slave-q03f krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:27:59.924618 22757 init.cc:377] Logged in from keytab as kudu/127.31.214.126@KRBTEST.COM (short username kudu)
I20260928 06:27:59.925091 22757 webserver.cc:460] Webserver started at http://127.31.214.126:36749/ using document root <none> and password file <none>
I20260928 06:27:59.925340 22757 fs_manager.cc:359] Metadata directory not provided
I20260928 06:27:59.925413 22757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:27:59.925570 22757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:27:59.926801 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/data/instance:
uuid: "105b93533906420eaccd354f52af1f35"
format_stamp: "Formatted at 2026-09-28 06:27:59 on dist-test-slave-q03f"
server_key: "ed29877e22dff30724e160ab0ab908f6"
server_key_iv: "fd9de9f80b8463310cfab80eccf1d34a"
server_key_version: "encryptionkey@0"
I20260928 06:27:59.927192 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/wal/instance:
uuid: "105b93533906420eaccd354f52af1f35"
format_stamp: "Formatted at 2026-09-28 06:27:59 on dist-test-slave-q03f"
server_key: "ed29877e22dff30724e160ab0ab908f6"
server_key_iv: "fd9de9f80b8463310cfab80eccf1d34a"
server_key_version: "encryptionkey@0"
I20260928 06:27:59.928709 22757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:27:59.929651 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:27:59.929931 22757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:27:59.930110 22757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/wal
uuid: "105b93533906420eaccd354f52af1f35"
format_stamp: "Formatted at 2026-09-28 06:27:59 on dist-test-slave-q03f"
server_key: "ed29877e22dff30724e160ab0ab908f6"
server_key_iv: "fd9de9f80b8463310cfab80eccf1d34a"
server_key_version: "encryptionkey@0"
I20260928 06:27:59.930200 22757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:27:59.944957 22757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:27:59.947449 22757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:27:59.947634 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:27:59.952409 22757 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:34711
I20260928 06:27:59.952452 22824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:34711 every 8 connection(s)
I20260928 06:27:59.952903 22757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/data/info.pb
I20260928 06:27:59.954028 22825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:27:59.954012 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 22757
I20260928 06:27:59.954126 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-0/wal/instance
I20260928 06:27:59.954346 32601 external_mini_cluster.cc:1428] Setting key c703ad5408f5d92d0ecb4a81209322dc
I20260928 06:27:59.956485 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.214.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.125@KRBTEST.COM" created.
Sep 28 06:27:59 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:27:59.970005 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:59.957255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:34711 (local address 127.0.0.1:41658)
0928 06:27:59.957487 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:59.957530 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:27:59.958232 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:27:59.958614 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:27:59.958617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:27:59.958890 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:27:59.959059 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:59.959063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:59.960807 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:59.960809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:59.961569 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:27:59.961571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:27:59.962382 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:27:59.962392 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:27:59.962467 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:59.963129 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:27:59.963138 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:27:59.966293 (+ 3155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:27:59.968184 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:59.968190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:59.968193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:59.968461 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:59.968736 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:27:59.968737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:27:59.968737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:27:59.968856 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:27:59.969259 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:27:59.969267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:27:59.969356 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 06:27:59.969468 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:27:59.969687 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260928 06:27:59.970053 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:59.957465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.31.214.126:34711)
0928 06:27:59.957713 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 06:27:59.957718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:27:59.957755 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:27:59.958457 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:27:59.958460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:27:59.958484 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:27:59.958598 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:27:59.959208 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:59.960695 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:59.961668 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:27:59.962255 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:27:59.962385 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:27:59.966390 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:27:59.966397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:27:59.966398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:27:59.966405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:27:59.968069 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:59.968559 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:59.968562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:59.968562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:59.968652 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:27:59.968943 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:27:59.968944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:27:59.968944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:27:59.969174 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:27:59.969253 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:27:59.969580 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 06:27:59.969782 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":214,"thread_start_us":72,"threads_started":1}
I20260928 06:27:59.970932 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:27:59.971345 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:59.971222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:42339 (local address 127.0.0.1:50396)
0928 06:27:59.971276 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:59.971325 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 06:27:59.971489 22774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
I20260928 06:27:59.971967 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:27:59.971862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:42339 (local address 127.0.0.1:50404)
0928 06:27:59.971913 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:27:59.971942 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:27:59.972074 22825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.125:42339: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab.
Entry for principal kudu/127.31.214.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab.
I20260928 06:27:59.985103 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.125:42339
--webserver_interface=127.31.214.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab
--principal=kudu/127.31.214.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:34711,127.31.214.125:42339,127.31.214.124:38913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:28:00.021337 22825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } attempt: 1
I20260928 06:28:00.021907 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.021759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:42339 (local address 127.0.0.1:50412)
0928 06:28:00.021827 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.021869 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:28:00.022332 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.022230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:42339 (local address 127.0.0.1:50426)
0928 06:28:00.022276 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.022301 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:28:00.022430 22825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.125:42339: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
W20260928 06:28:00.075443 22832 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:28:00.075729 22832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:00.075762 22832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:00.075807 22832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:00.077101 22832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:00.077188 22832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:00.077211 22832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:00.077232 22832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:00.077260 22832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:00.078989 22832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:34711,127.31.214.125:42339,127.31.214.124:38913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.125.keytab
--principal=kudu/127.31.214.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.125:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.214.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:00.079315 22832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:00.079573 22832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:00.085319 22840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:00.085338 22838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:00.085366 22837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:00.085739 22832 server_base.cc:1034] running on GCE node
I20260928 06:28:00.086107 22832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:00.086403 22832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:00.087599 22832 hybrid_clock.cc:648] HybridClock initialized: now 1790576880087576 us; error 46 us; skew 500 ppm
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:00.090579 22825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } attempt: 2
I20260928 06:28:00.090802 22832 init.cc:377] Logged in from keytab as kudu/127.31.214.125@KRBTEST.COM (short username kudu)
I20260928 06:28:00.091142 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.090973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:42339 (local address 127.0.0.1:50434)
0928 06:28:00.091052 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.091099 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:28:00.091490 22832 webserver.cc:460] Webserver started at http://127.31.214.125:33011/ using document root <none> and password file <none>
I20260928 06:28:00.091588 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.091461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:42339 (local address 127.0.0.1:50448)
0928 06:28:00.091520 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.091555 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:28:00.091786 22825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.125:42339: Network error: Client connection negotiation failed: client connection to 127.31.214.125:42339: connect: Connection refused (error 111)
I20260928 06:28:00.091830 22832 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:00.091908 22832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:00.092068 22832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:00.093298 22832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/data/instance:
uuid: "67bbfa163303469693074bcce103a3b6"
format_stamp: "Formatted at 2026-09-28 06:28:00 on dist-test-slave-q03f"
server_key: "8e8e18b34d648846fe3a70fd3d4c7f6e"
server_key_iv: "aa477c4c51889cfb0ac4b8008a8ef103"
server_key_version: "encryptionkey@0"
I20260928 06:28:00.093739 22832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/wal/instance:
uuid: "67bbfa163303469693074bcce103a3b6"
format_stamp: "Formatted at 2026-09-28 06:28:00 on dist-test-slave-q03f"
server_key: "8e8e18b34d648846fe3a70fd3d4c7f6e"
server_key_iv: "aa477c4c51889cfb0ac4b8008a8ef103"
server_key_version: "encryptionkey@0"
I20260928 06:28:00.095662 22832 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 06:28:00.096537 22847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:00.096927 22832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:00.097075 22832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/wal
uuid: "67bbfa163303469693074bcce103a3b6"
format_stamp: "Formatted at 2026-09-28 06:28:00 on dist-test-slave-q03f"
server_key: "8e8e18b34d648846fe3a70fd3d4c7f6e"
server_key_iv: "aa477c4c51889cfb0ac4b8008a8ef103"
server_key_version: "encryptionkey@0"
I20260928 06:28:00.097174 22832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:00.115758 22832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:00.117869 22832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:00.118108 22832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:00.122944 22832 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.125:42339
I20260928 06:28:00.122982 22899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.125:42339 every 8 connection(s)
I20260928 06:28:00.123415 22832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/data/info.pb
I20260928 06:28:00.124343 22900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:00.126137 22900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:28:00.131109 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 22832
I20260928 06:28:00.131243 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-1/wal/instance
I20260928 06:28:00.131457 32601 external_mini_cluster.cc:1428] Setting key a4a43299674ea26cd4105ad717665544
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:28:00.137492 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.126991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.31.214.126:34711)
0928 06:28:00.127209 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.127211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.127474 (+ 263us) server_negotiation.cc:407] Connection header received
0928 06:28:00.128000 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.128002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.128015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.128079 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.128765 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.129291 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.130462 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.130688 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.130786 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.133872 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.133873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.133873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.133882 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.135818 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.136380 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.136381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.136381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.136435 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.136754 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.136754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.136755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.136878 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.136935 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.137341 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.137421 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260928 06:28:00.137801 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.126885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:34711 (local address 127.0.0.1:41660)
0928 06:28:00.127252 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.127298 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:00.127867 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:00.128164 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:00.128168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:00.128455 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:00.128661 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.128666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.129443 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.129444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.130342 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.130345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.130796 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.130797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.130960 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.131653 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:00.131676 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:00.133743 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:00.135929 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.135939 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.135941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.136283 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.136500 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.136501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.136502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.136669 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.136952 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:00.136960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:00.137160 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 06:28:00.137296 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 06:28:00.137503 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260928 06:28:00.138554 22900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.214.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.214.124@KRBTEST.COM" created.
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:28:00.147686 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.138907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:42339 (local address 127.0.0.1:50460)
0928 06:28:00.138961 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.138973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:00.139016 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:00.139412 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:00.139414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:00.139579 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:00.139633 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.139635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.141031 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.141032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.141575 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.141577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.142156 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.142157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.142222 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.142661 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:00.142666 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:00.144659 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:00.146573 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.146574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.146574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.146740 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.146975 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.146976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.146976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.147031 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.147334 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:00.147335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:00.147374 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:28:00.147437 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:28:00.147543 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:28:00.147710 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.139015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.31.214.125:42339)
0928 06:28:00.139205 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.139211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.139237 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:28:00.139262 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.139264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.139280 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.139374 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.139728 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.140877 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.141664 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.142058 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.142125 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.144765 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.144774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.144774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.144782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.146470 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.146836 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.146840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.146840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.146896 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.147103 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.147104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.147105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.147256 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.147345 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.147473 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.147599 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":156,"thread_start_us":112,"threads_started":1}
I20260928 06:28:00.148324 22900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:28:00.148739 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.148613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47318)
0928 06:28:00.148664 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.148713 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 06:28:00.148916 22849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
I20260928 06:28:00.149461 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.149337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47328)
0928 06:28:00.149403 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.149438 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 06:28:00.149605 22900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:38913: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.214.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab.
Entry for principal kudu/127.31.214.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab.
I20260928 06:28:00.162202 32601 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDYGmsK/build/release/bin/kudu
/tmp/dist-test-taskDYGmsK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.124:38913
--webserver_interface=127.31.214.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab
--principal=kudu/127.31.214.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.214.84:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.214.126:34711,127.31.214.125:42339,127.31.214.124:38913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:28:00.182953 22825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } attempt: 3
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:28:00.190687 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.183378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:42339 (local address 127.0.0.1:50474)
0928 06:28:00.183452 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.183463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:00.183550 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:00.183701 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:00.183702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:00.183872 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:00.183940 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.183942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.184670 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.184671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.185214 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.185215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.185619 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.185621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.185703 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.186257 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:00.186262 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:00.187749 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:00.189459 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.189460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.189461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.189650 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.189932 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.189933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.189933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.190037 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.190317 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:00.190318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:00.190368 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:28:00.190459 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:28:00.190575 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:28:00.190680 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.183457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.31.214.125:42339)
0928 06:28:00.183528 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.183530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.183542 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:28:00.183619 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.183620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.183633 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.183686 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.184030 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.184563 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.185302 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.185534 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.185615 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.187854 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.187855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.187855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.187864 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.189351 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.189747 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.189749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.189749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.189860 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.190123 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.190124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.190125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.190248 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.190326 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.190498 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.190577 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:28:00.191335 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:28:00.191802 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.191667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47342)
0928 06:28:00.191730 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.191770 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:28:00.192140 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.192042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47344)
0928 06:28:00.192090 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.192119 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:28:00.192238 22825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:38913: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
I20260928 06:28:00.198879 22900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } attempt: 1
I20260928 06:28:00.199373 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.199233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47350)
0928 06:28:00.199300 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.199344 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:28:00.199790 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.199686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47364)
0928 06:28:00.199735 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.199762 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:28:00.199884 22900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:38913: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
I20260928 06:28:00.223402 22825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } attempt: 1
I20260928 06:28:00.223974 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.223802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47370)
0928 06:28:00.223874 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.223925 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:28:00.224375 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.224281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47374)
0928 06:28:00.224327 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.224355 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:28:00.224478 22825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:38913: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
W20260928 06:28:00.249917 22907 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:28:00.250226 22907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:28:00.250244 22907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:28:00.250265 22907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:28:00.251504 22907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:28:00.251549 22907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:28:00.251562 22907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:28:00.251575 22907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:28:00.251587 22907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:28:00.253160 22907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.214.84:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.214.126:34711,127.31.214.125:42339,127.31.214.124:38913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/krb5kdc/kudu_127.31.214.124.keytab
--principal=kudu/127.31.214.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.214.124:38913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.214.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:40:12 UTC on ccycloud.quasar-qehodb.root.comops.site
build id 4676664
I20260928 06:28:00.253441 22907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:28:00.253700 22907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:28:00.259199 22912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:00.259348 22907 server_base.cc:1034] running on GCE node
W20260928 06:28:00.259244 22913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:28:00.259205 22915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:28:00.259842 22907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:28:00.260196 22907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:28:00.261380 22907 hybrid_clock.cc:648] HybridClock initialized: now 1790576880261363 us; error 39 us; skew 500 ppm
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:28:00.264123 22907 init.cc:377] Logged in from keytab as kudu/127.31.214.124@KRBTEST.COM (short username kudu)
I20260928 06:28:00.264676 22907 webserver.cc:460] Webserver started at http://127.31.214.124:39963/ using document root <none> and password file <none>
I20260928 06:28:00.264886 22907 fs_manager.cc:359] Metadata directory not provided
I20260928 06:28:00.264930 22907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:28:00.265027 22907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:28:00.266188 22907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/data/instance:
uuid: "a0960fcb3d8845a5b816c294e9738732"
format_stamp: "Formatted at 2026-09-28 06:28:00 on dist-test-slave-q03f"
server_key: "fc2a897ad049e0ab9c3ce8bb235acd27"
server_key_iv: "b9a56fc35a2ee17135727a4a57f3a2ad"
server_key_version: "encryptionkey@0"
I20260928 06:28:00.266521 22907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/wal/instance:
uuid: "a0960fcb3d8845a5b816c294e9738732"
format_stamp: "Formatted at 2026-09-28 06:28:00 on dist-test-slave-q03f"
server_key: "fc2a897ad049e0ab9c3ce8bb235acd27"
server_key_iv: "b9a56fc35a2ee17135727a4a57f3a2ad"
server_key_version: "encryptionkey@0"
I20260928 06:28:00.268029 22907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:00.268054 22900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } attempt: 2
I20260928 06:28:00.268563 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.268405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47382)
0928 06:28:00.268480 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.268526 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:28:00.268999 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.268885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47388)
0928 06:28:00.268941 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.268968 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:28:00.269129 22900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.214.124:38913: Network error: Client connection negotiation failed: client connection to 127.31.214.124:38913: connect: Connection refused (error 111)
I20260928 06:28:00.269178 22922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:28:00.269339 22907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:28:00.269429 22907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/data,/tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/wal
uuid: "a0960fcb3d8845a5b816c294e9738732"
format_stamp: "Formatted at 2026-09-28 06:28:00 on dist-test-slave-q03f"
server_key: "fc2a897ad049e0ab9c3ce8bb235acd27"
server_key_iv: "b9a56fc35a2ee17135727a4a57f3a2ad"
server_key_version: "encryptionkey@0"
I20260928 06:28:00.269488 22907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:28:00.282311 22907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:28:00.284441 22907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:28:00.284641 22907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:28:00.289156 22907 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.124:38913
I20260928 06:28:00.289194 22974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.124:38913 every 8 connection(s)
I20260928 06:28:00.289578 22907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/data/info.pb
I20260928 06:28:00.290575 22975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:28:00.292315 22975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:28:00.297523 32601 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDYGmsK/build/release/bin/kudu as pid 22907
I20260928 06:28:00.297621 32601 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDYGmsK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790576570121568-32601-0/minicluster-data/master-2/wal/instance
I20260928 06:28:00.297857 32601 external_mini_cluster.cc:1428] Setting key d600a350fa63ca81b616c2910970e70d
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:28:00.299628 22825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } attempt: 2
I20260928 06:28:00.303278 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.293165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.31.214.126:34711)
0928 06:28:00.293240 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.293242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.293429 (+ 187us) server_negotiation.cc:407] Connection header received
0928 06:28:00.293898 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.293899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.293997 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.294078 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.294727 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.295300 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.296424 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.296649 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.296719 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.299497 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.299498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.299498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.299506 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.301341 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.301861 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.301862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.301862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.301926 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.302231 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.302233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.302233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.302387 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.302459 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.303088 (+ 629us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.303165 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:28:00.303507 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.293082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.126:34711 (local address 127.0.0.1:41676)
0928 06:28:00.293267 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.293314 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:00.293775 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:00.294149 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:00.294153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:00.294422 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:00.294628 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.294632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.295413 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.295414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.296336 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.296339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.296738 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.296739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.296843 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.297350 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:00.297358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:00.299398 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:00.301421 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.301429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.301432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.301757 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.302008 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.302009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.302009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.302153 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.302465 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:00.302472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:00.302637 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 06:28:00.303042 (+ 405us) client_negotiation.cc:240] Negotiation successful
0928 06:28:00.303262 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260928 06:28:00.304216 22975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576879, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.126@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
I20260928 06:28:00.310010 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.300221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47416)
0928 06:28:00.300631 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.300644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:00.300685 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:00.301009 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:00.301010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:00.301234 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:00.301270 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.301272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.302266 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.302268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.303050 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.303054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.303511 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.303512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.303582 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.304167 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:00.304172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:00.306367 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:00.308701 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.308702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.308703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.308955 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.309269 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.309270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.309271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.309336 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.309646 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:00.309647 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:00.309700 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:28:00.309756 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:28:00.309864 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388}
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:28:00.310083 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.298333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.31.214.124:38913)
0928 06:28:00.298499 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.298504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.298530 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:28:00.298620 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.298622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.298641 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.298707 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.299513 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.300638 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.301791 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.302287 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.302352 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.306346 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.306354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.306355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.306362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.308441 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.308865 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.308870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.308871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.308985 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.309211 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.309212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.309213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.309436 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.309616 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.309699 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.309964 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260928 06:28:00.310936 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.300315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.31.214.124:38913)
0928 06:28:00.300887 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.300889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.300895 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:28:00.300919 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.300920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.300927 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.301004 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.301396 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.302168 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.303152 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.303432 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.303502 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.306459 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.306460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.306460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.306466 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.308580 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.309070 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.309071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.309072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.309149 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.309409 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.309410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.309410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.309552 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.310570 (+ 1018us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.310616 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.310812 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":514,"threads_started":1}
I20260928 06:28:00.312862 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35: Bootstrap starting.
I20260928 06:28:00.313616 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.304564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.31.214.125:42339)
0928 06:28:00.304613 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.304615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.304733 (+ 118us) server_negotiation.cc:407] Connection header received
0928 06:28:00.304796 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.304798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.304811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.304872 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.305118 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.305856 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.308018 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.308272 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.308379 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.310450 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.310452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.310452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.310460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.312254 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.312662 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.312664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.312664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.312735 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.312988 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.312990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.312990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.313128 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.313173 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.313420 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.313522 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:28:00.313812 22825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:00.314008 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.304482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.125:42339 (local address 127.0.0.1:50484)
0928 06:28:00.304649 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.304661 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:00.304727 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:00.304874 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:00.304875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:00.304993 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:00.305045 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.305046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.306459 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.306459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.307886 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.307889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.308380 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.308381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.308464 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.308930 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:00.308935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:00.310372 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:00.312360 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.312361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.312361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.312577 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.312831 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.312832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.312832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.312900 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.313229 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:00.313230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:00.313281 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:28:00.313755 (+ 474us) client_negotiation.cc:240] Negotiation successful
0928 06:28:00.313863 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260928 06:28:00.314224 22825 log.cc:824] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:00.314509 22975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:28:00.315392 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35: No bootstrap required, opened a new log
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.126@KRBTEST.COM
I20260928 06:28:00.317059 22825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } }
I20260928 06:28:00.317211 22825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105b93533906420eaccd354f52af1f35, State: Initialized, Role: FOLLOWER
I20260928 06:28:00.317313 22825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } }
I20260928 06:28:00.317713 22825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:00.318275 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.318404 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:00.319669 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.311378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.31.214.126:34711)
0928 06:28:00.311422 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.311424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.311432 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:00.311450 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.311451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.311461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.311509 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.312053 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.312681 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.313431 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.313639 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.313719 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.316348 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.316350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.316351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.316358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.318067 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.318574 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.318576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.318576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.318650 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.318888 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.318889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.318890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.319045 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.319185 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.319458 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.319576 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:28:00.321702 22993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:28:00.321815 22993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.124@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
I20260928 06:28:00.326136 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.314826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47432)
0928 06:28:00.314937 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.314947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:00.314985 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:00.318167 (+ 3182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:00.318169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:00.318325 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:00.318451 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.318454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.319268 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.319269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.319995 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.319999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.320486 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.320487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.320538 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.320992 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:00.320998 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:00.322685 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:00.324893 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.324894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.324895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.325144 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.325358 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.325358 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.325359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.325419 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.325732 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:00.325733 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:00.325774 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:28:00.325845 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:28:00.325980 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 06:28:00.326376 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.314901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.31.214.124:38913)
0928 06:28:00.318036 (+ 3135us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.318038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.318048 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:00.318074 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.318075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.318084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.318149 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.318578 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.319172 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.320092 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.320376 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.320439 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.322821 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.322823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.322823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.322831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.324785 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.325232 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.325232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.325233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.325284 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.325500 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.325501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.325501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.325625 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.325687 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.325891 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.326295 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":309,"server-negotiator.queue_time_us":40}
I20260928 06:28:00.321045 22757 master_runner.cc:186] Error getting master registration for 127.31.214.125:42339: OK, instance_id { permanent_uuid: "67bbfa163303469693074bcce103a3b6" instance_seqno: 1790576880121555 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:28:00.327945 22757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.214.125@KRBTEST.COM
I20260928 06:28:00.329752 22975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732: Bootstrap starting.
I20260928 06:28:00.330654 22975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:00.330986 22975 log.cc:824] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:00.331974 22975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732: No bootstrap required, opened a new log
I20260928 06:28:00.332773 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.320485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.31.214.125:42339)
0928 06:28:00.320526 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.320528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.320538 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:00.320563 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.320564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.320574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.320628 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.321522 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.322747 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.324908 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.325213 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.325281 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.328839 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.328841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.328841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.328850 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.330914 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.331426 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.331427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.331427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.331503 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.331696 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.331697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.331698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.331836 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.332112 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.332517 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.332637 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:28:00.334277 22975 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } }
I20260928 06:28:00.334476 22975 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0960fcb3d8845a5b816c294e9738732, State: Initialized, Role: FOLLOWER
I20260928 06:28:00.334581 22975 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } }
I20260928 06:28:00.334794 32601 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:28:00.334934 22975 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:00.334877 22996 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.334980 22996 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:00.335819 22799 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "105b93533906420eaccd354f52af1f35"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41682
I20260928 06:28:00.336030 22799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:28:00.336098 22799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [term 0 FOLLOWER]: Advancing to term 1
W20260928 06:28:00.336998 23007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:28:00.337046 23007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:28:00.337204 22799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } }
I20260928 06:28:00.337478 22799 leader_election.cc:288] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [CANDIDATE]: Term 1 election: Requested vote from peers 67bbfa163303469693074bcce103a3b6 (127.31.214.125:42339), a0960fcb3d8845a5b816c294e9738732 (127.31.214.124:38913)
I20260928 06:28:00.338294 22949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "105b93533906420eaccd354f52af1f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a0960fcb3d8845a5b816c294e9738732"
I20260928 06:28:00.338411 22949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:00.339324 22949 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 105b93533906420eaccd354f52af1f35 in term 1.
I20260928 06:28:00.339521 22774 leader_election.cc:302] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105b93533906420eaccd354f52af1f35, a0960fcb3d8845a5b816c294e9738732; no voters:
I20260928 06:28:00.339669 22982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:28:00.339843 22982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [term 1 LEADER]: Becoming Leader. State: Replica: 105b93533906420eaccd354f52af1f35, State: Running, Role: LEADER
I20260928 06:28:00.339947 22982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } }
I20260928 06:28:00.340389 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105b93533906420eaccd354f52af1f35. Latest consensus state: current_term: 1 leader_uuid: "105b93533906420eaccd354f52af1f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.340503 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:00.339183 22907 master_runner.cc:186] Error getting master registration for 127.31.214.125:42339: OK, instance_id { permanent_uuid: "67bbfa163303469693074bcce103a3b6" instance_seqno: 1790576880121555 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:28:00.341434 22907 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:28:00.342633 23015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:28:00.342840 23015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:28:00.345142 22949 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [term 1 FOLLOWER]: Refusing update from remote peer 105b93533906420eaccd354f52af1f35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:00.345538 23013 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:00.346870 22996 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105b93533906420eaccd354f52af1f35. Latest consensus state: current_term: 1 leader_uuid: "105b93533906420eaccd354f52af1f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.347473 22996 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:00.348280 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "105b93533906420eaccd354f52af1f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.348363 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:00.348548 22996 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "105b93533906420eaccd354f52af1f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.348640 22996 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:00.348971 23020 mvcc.cc:205] Tried to move back new op lower bound from 7334202901889712128 to 7334202901873106944. Current Snapshot: MvccSnapshot[applied={T|T < 7334202901889712128}]
I20260928 06:28:00.349826 23015 catalog_manager.cc:1347] Generated new cluster ID: 750f4cd7a28643cd83514732672e4ee8
I20260928 06:28:00.349879 23015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:28:00.359987 23015 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:28:00.360242 22900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } attempt: 3
I20260928 06:28:00.362071 23015 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 06:28:00 dist-test-slave-q03f krb5kdc[22744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790576880, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.214.125@KRBTEST.COM for kudu/127.31.214.124@KRBTEST.COM
I20260928 06:28:00.370988 22902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.360529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.214.124:38913 (local address 127.0.0.1:47450)
0928 06:28:00.360574 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:28:00.360584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:28:00.360618 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:28:00.360777 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:28:00.360777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:28:00.360951 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:28:00.360994 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.360996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.361916 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.361917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.362720 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:28:00.362723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.363361 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.363362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:28:00.363423 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.364103 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:28:00.364110 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:28:00.366528 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:28:00.369693 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.369695 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.369695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.369959 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.370225 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:28:00.370226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:28:00.370226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:28:00.370300 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:28:00.370632 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:28:00.370633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:28:00.370699 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 06:28:00.370754 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:28:00.370865 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:28:00.371780 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.360577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.31.214.124:38913)
0928 06:28:00.360615 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.360617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.360625 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:28:00.360684 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.360685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.360695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.360740 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.361062 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.361821 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.362897 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.363245 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.363303 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.366657 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.366659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.366659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.366682 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.369564 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.370055 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.370057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.370057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.370142 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.370373 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.370374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.370375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.370530 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.370588 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.371510 (+ 922us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.371615 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:00.373809 22900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6: Bootstrap starting.
I20260928 06:28:00.374780 23015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35: Generated new TSK 0
I20260928 06:28:00.375108 23015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:28:00.375757 22900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6: Neither blocks nor log segments found. Creating new log.
I20260928 06:28:00.376113 22900 log.cc:824] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:28:00.377358 22900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6: No bootstrap required, opened a new log
I20260928 06:28:00.378360 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.343903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.31.214.126:34711)
0928 06:28:00.343960 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.343962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.354207 (+ 10245us) server_negotiation.cc:407] Connection header received
0928 06:28:00.354272 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.354273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.354302 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.354412 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.357626 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.358444 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.362840 (+ 4396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.363127 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.363173 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.374050 (+ 10877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.374053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.374053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.374062 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.376101 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.376858 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.376860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.376860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.376926 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.377232 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.377233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.377233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.377384 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.377481 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.378151 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.378264 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:28:00.380280 22900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } }
I20260928 06:28:00.380756 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.344619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.31.214.125:42339)
0928 06:28:00.344669 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.344671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.354333 (+ 9662us) server_negotiation.cc:407] Connection header received
0928 06:28:00.357736 (+ 3403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.357738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.357752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.357783 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.360378 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.361081 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.363694 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.364030 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.364080 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.374464 (+ 10384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.374466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.374466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.374474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.376960 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.377731 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.377732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.377732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.377790 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.379917 (+ 2127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.379918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.379918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.380110 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.380192 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.380462 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.380550 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:28:00.380564 22900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bbfa163303469693074bcce103a3b6, State: Initialized, Role: FOLLOWER
I20260928 06:28:00.381320 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.344969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.31.214.124:38913)
0928 06:28:00.345009 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.345011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.358604 (+ 13593us) server_negotiation.cc:407] Connection header received
0928 06:28:00.358663 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.358664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.358676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.358707 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.359424 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.360174 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.365457 (+ 5283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.365779 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.365828 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.377414 (+ 11586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:28:00.377415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:28:00.377415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:28:00.377424 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:28:00.379227 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.380322 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.380323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.380324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.380386 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:28:00.380758 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:28:00.380759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:28:00.380760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:28:00.380910 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:28:00.381001 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.381149 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.381229 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:00.381726 22900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } }
I20260928 06:28:00.382467 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "105b93533906420eaccd354f52af1f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "67bbfa163303469693074bcce103a3b6"
I20260928 06:28:00.382493 22873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:28:00.382325 23025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.382555 22900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:28:00.382663 23025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:00.383500 22873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [term 1 FOLLOWER]: Refusing update from remote peer 105b93533906420eaccd354f52af1f35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:28:00.384147 22982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:28:00.384618 22874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 105b93533906420eaccd354f52af1f35 in term 1.
I20260928 06:28:00.388247 23042 catalog_manager.cc:1259] Loaded cluster ID: 750f4cd7a28643cd83514732672e4ee8
I20260928 06:28:00.388305 23042 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6: loading cluster ID for follower catalog manager: success
I20260928 06:28:00.388648 23025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105b93533906420eaccd354f52af1f35. Latest consensus state: current_term: 1 leader_uuid: "105b93533906420eaccd354f52af1f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.388798 23025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:00.388875 23025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "105b93533906420eaccd354f52af1f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.388964 23025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:28:00.389529 23042 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6: acquiring CA information for follower catalog manager: success
I20260928 06:28:00.389904 23042 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67bbfa163303469693074bcce103a3b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:28:00.416168 22829 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.414390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.31.214.126:34711)
0928 06:28:00.414449 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.414452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.415369 (+ 917us) server_negotiation.cc:407] Connection header received
0928 06:28:00.415482 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.415484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.415496 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.415530 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.416085 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41700: BlockingRecv error: recv got EOF from 127.0.0.1:41700 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 06:28:00.417071 22904 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.414763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.31.214.125:42339)
0928 06:28:00.414811 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.414814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.416413 (+ 1599us) server_negotiation.cc:407] Connection header received
0928 06:28:00.416516 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.416517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.416529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.416560 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.417002 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50508: BlockingRecv error: recv got EOF from 127.0.0.1:50508 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 06:28:00.417932 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.415072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.31.214.124:38913)
0928 06:28:00.415115 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.415117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.417334 (+ 2217us) server_negotiation.cc:407] Connection header received
0928 06:28:00.417485 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.417487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.417498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.417530 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.417867 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47452: BlockingRecv error: recv got EOF from 127.0.0.1:47452 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:00.423542 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.419747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.31.214.125:42339)
0928 06:28:00.419784 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.419786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.420020 (+ 234us) server_negotiation.cc:407] Connection header received
0928 06:28:00.420137 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.420138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.420191 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.420241 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:00.420449 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.421333 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.422546 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.422849 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.422891 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.423128 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:00.423275 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:00.423328 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.423440 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.423460 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:28:00.427052 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.426198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.31.214.124:38913)
0928 06:28:00.426240 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.426242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.426470 (+ 228us) server_negotiation.cc:407] Connection header received
0928 06:28:00.426582 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.426583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.426605 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.426651 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.426992 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47468: BlockingRecv error: recv got EOF from 127.0.0.1:47468 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:00.432899 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.429340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.31.214.125:42339)
0928 06:28:00.429381 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.429383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.429599 (+ 216us) server_negotiation.cc:407] Connection header received
0928 06:28:00.429700 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.429701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.429710 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.429737 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:00.429889 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.430834 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.432046 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.432371 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.432413 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.432642 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:00.432679 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:00.432705 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.432800 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.432820 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 06:28:00.442836 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.438347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.31.214.124:38913)
0928 06:28:00.438393 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.438396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.442177 (+ 3781us) server_negotiation.cc:407] Connection header received
0928 06:28:00.442300 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.442301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.442316 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.442348 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.442772 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47472: BlockingRecv error: recv got EOF from 127.0.0.1:47472 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:00.449573 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.446326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.31.214.125:42339)
0928 06:28:00.446372 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.446374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.446603 (+ 229us) server_negotiation.cc:407] Connection header received
0928 06:28:00.446710 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.446711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.446722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.446752 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:00.446938 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.447659 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.448760 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.449031 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.449077 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.449280 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:00.449314 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:00.449342 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.449462 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.449485 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:28:00.453056 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.452107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.31.214.124:38913)
0928 06:28:00.452147 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.452149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.452425 (+ 276us) server_negotiation.cc:407] Connection header received
0928 06:28:00.452535 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.452536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.452549 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.452591 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.452994 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47474: BlockingRecv error: recv got EOF from 127.0.0.1:47474 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:00.461936 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.458515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.31.214.125:42339)
0928 06:28:00.458576 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.458579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.458858 (+ 279us) server_negotiation.cc:407] Connection header received
0928 06:28:00.458967 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.458968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.458980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.459010 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:00.459208 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.459956 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.461086 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.461371 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.461419 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.461654 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:00.461693 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:00.461722 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.461832 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.461853 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:28:00.465245 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.464332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.31.214.124:38913)
0928 06:28:00.464377 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.464379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.464604 (+ 225us) server_negotiation.cc:407] Connection header received
0928 06:28:00.464711 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.464712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.464725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.464758 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.465188 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47488: BlockingRecv error: recv got EOF from 127.0.0.1:47488 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:28:00.478222 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.474683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.31.214.125:42339)
0928 06:28:00.474729 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.474731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.475027 (+ 296us) server_negotiation.cc:407] Connection header received
0928 06:28:00.475141 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.475142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.475154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.475186 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:00.475375 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.476133 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.477318 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.477623 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.477670 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.477884 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:00.477920 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:00.477983 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.478117 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.478138 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:28:00.481600 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.480690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.31.214.124:38913)
0928 06:28:00.480732 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.480734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.481004 (+ 270us) server_negotiation.cc:407] Connection header received
0928 06:28:00.481108 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.481110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.481123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.481153 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.481524 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47490: BlockingRecv error: recv got EOF from 127.0.0.1:47490 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:00.502784 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.499120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.31.214.125:42339)
0928 06:28:00.499170 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.499173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.499494 (+ 321us) server_negotiation.cc:407] Connection header received
0928 06:28:00.499609 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.499610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.499622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.499650 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:00.499830 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.500630 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.501855 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.502208 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.502261 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.502487 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:00.502523 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:00.502551 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.502676 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.502699 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:28:00.506332 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.505341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.31.214.124:38913)
0928 06:28:00.505388 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.505390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.505683 (+ 293us) server_negotiation.cc:407] Connection header received
0928 06:28:00.505800 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.505801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.505814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.505846 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.506271 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47504: BlockingRecv error: recv got EOF from 127.0.0.1:47504 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:28:00.542970 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.539970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.31.214.125:42339)
0928 06:28:00.540033 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.540036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.540165 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:28:00.540262 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.540263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.540275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.540354 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:00.540546 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.541114 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.542147 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.542370 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.542476 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.542651 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:00.542697 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:00.542766 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.542828 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.542851 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 06:28:00.545684 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.544956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.31.214.124:38913)
0928 06:28:00.545001 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.545003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.545052 (+ 49us) server_negotiation.cc:407] Connection header received
0928 06:28:00.545160 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.545161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.545171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.545221 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.545606 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47520: BlockingRecv error: recv got EOF from 127.0.0.1:47520 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:00.613623 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.611001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.31.214.125:42339)
0928 06:28:00.611065 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.611066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.611251 (+ 185us) server_negotiation.cc:407] Connection header received
0928 06:28:00.611343 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.611345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.611363 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.611430 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:00.611583 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.612085 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.612904 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.613107 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.613187 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.613338 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:00.613381 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:00.613442 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.613508 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.613528 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 06:28:00.616532 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.615656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.31.214.124:38913)
0928 06:28:00.615777 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.615780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.615871 (+ 91us) server_negotiation.cc:407] Connection header received
0928 06:28:00.616035 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.616036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.616047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.616122 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.616470 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47528: BlockingRecv error: recv got EOF from 127.0.0.1:47528 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 06:28:00.748965 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:00.746031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.31.214.125:42339)
0928 06:28:00.746150 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.746153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.746285 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:28:00.746398 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.746399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.746412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.746492 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:00.746719 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.747271 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.748165 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:00.748384 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:00.748480 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:00.748661 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:00.748709 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:00.748779 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:00.748842 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:28:00.748865 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 06:28:00.752077 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:00.751241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.31.214.124:38913)
0928 06:28:00.751300 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:00.751302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:00.751387 (+ 85us) server_negotiation.cc:407] Connection header received
0928 06:28:00.751533 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:00.751534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:00.751545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:00.751596 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:00.751986 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47540: BlockingRecv error: recv got EOF from 127.0.0.1:47540 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:28:00.827322 22982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "105b93533906420eaccd354f52af1f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105b93533906420eaccd354f52af1f35" member_type: VOTER last_known_addr { host: "127.31.214.126" port: 34711 } } peers { permanent_uuid: "67bbfa163303469693074bcce103a3b6" member_type: VOTER last_known_addr { host: "127.31.214.125" port: 42339 } } peers { permanent_uuid: "a0960fcb3d8845a5b816c294e9738732" member_type: VOTER last_known_addr { host: "127.31.214.124" port: 38913 } } }
I20260928 06:28:00.827489 22982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105b93533906420eaccd354f52af1f35 [sys.catalog]: This master's current role is: LEADER
I20260928 06:28:01.012122 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:01.009589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.31.214.125:42339)
0928 06:28:01.009672 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:01.009674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:01.009684 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:01.009719 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:01.009720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:01.009735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:01.009803 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:01.009985 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.010492 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.011294 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.011578 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.011661 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:01.011834 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:01.011879 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:01.011938 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:01.012002 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:28:01.012024 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 06:28:01.014816 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:28:01.014060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.31.214.124:38913)
0928 06:28:01.014136 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:01.014138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:01.014232 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:28:01.014302 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:01.014303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:01.014315 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:01.014365 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:28:01.014745 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47552: BlockingRecv error: recv got EOF from 127.0.0.1:47552 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:28:01.337543 23007 catalog_manager.cc:1259] Loaded cluster ID: 750f4cd7a28643cd83514732672e4ee8
I20260928 06:28:01.337625 23007 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732: loading cluster ID for follower catalog manager: success
I20260928 06:28:01.338557 23007 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732: acquiring CA information for follower catalog manager: success
I20260928 06:28:01.338845 23007 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0960fcb3d8845a5b816c294e9738732: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:28:01.531011 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:01.528104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.31.214.125:42339)
0928 06:28:01.528219 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:01.528221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:01.528231 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:28:01.528311 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:01.528312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:01.528322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:01.528388 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:01.528698 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.529251 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.530192 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.530396 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.530463 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:01.530655 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:01.530691 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:01.530739 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:01.530834 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 06:28:01.530862 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260928 06:28:01.536609 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:01.533132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.31.214.124:38913)
0928 06:28:01.533272 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:01.533274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:01.533283 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:28:01.533307 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:01.533307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:01.533360 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:01.533442 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:01.533844 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.534507 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.535597 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.535856 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.535920 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:01.536171 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:01.536328 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:01.536448 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:01.536473 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:28:01.536498 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260928 06:28:01.542028 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:01.538992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.31.214.124:38913)
0928 06:28:01.539033 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:01.539035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:01.539107 (+ 72us) server_negotiation.cc:407] Connection header received
0928 06:28:01.539164 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:01.539165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:01.539175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:01.539222 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:01.539396 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.540269 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.541118 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.541361 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.541438 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:01.541650 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:01.541686 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:01.541717 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:01.541839 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 06:28:01.541865 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:28:01.542450 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:01.538845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.31.214.125:42339)
0928 06:28:01.538901 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:01.538903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:01.539025 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:28:01.539062 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:01.539063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:01.539076 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:01.539149 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:01.539351 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.540220 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.541415 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.541741 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.541820 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:01.542024 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:01.542075 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:01.542135 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:01.542318 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 06:28:01.542350 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:28:01.542572 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:28:01.538758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.31.214.126:34711)
0928 06:28:01.538892 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:28:01.538894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:28:01.538906 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:28:01.538958 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:28:01.538959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:28:01.539064 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:28:01.539211 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:28:01.539364 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.539893 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.540881 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:28:01.541301 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:28:01.541448 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:28:01.541679 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:28:01.541809 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:28:01.541918 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:28:01.542399 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 06:28:01.542469 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 06:28:01.545104 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 22757
I20260928 06:28:01.551131 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 22832
I20260928 06:28:01.556988 32601 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDYGmsK/build/release/bin/kudu with pid 22907
2026-09-28T06:28:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2959 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5889 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (311437 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS